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ABSTRACT 
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51C SMCT , in accordance with Army training doctrine, has helped subordinate units and 
individual CCOs within the ECC to focus their training efforts prior to deployment. This 
body of knowledge is a significant step in standardizing the pre-deployment training for 
CCOs and preparing them for their mission once deployed. To supplement this training, 
an appropriate performance-assessment tool is needed to measure performance of these 
36 tasks. Currently, the ECC lacks a standardized method of measuring individual skill 
and task proficiency in its CCOs. This research developed the questions for a written 
proficiency test based on the 36 tasks listed in the SMCT in accordance with conditions, 
standards, and performance steps for each of the 36 tasks in the SMCT. The research 
team vetted each of the scenarios and accompanying questions through experienced CCO 
focus groups within the ECC. Upon completion of the test, the research team provided a 
written Contracting Officer Proficiency-Assessment Test, encompassing over 1,100 
questions covering each of the 36 tasks listed in the SMCT , to the Expeditionary 
Contracting Command. 


v 



THIS PAGE INTENTIONALLY LEFT BLANK 


vi 



TABLE OF CONTENTS 


I. INTRODUCTION.1 

A. BACKGROUND.1 

B. PROBLEM STATEMENT.2 

C. RESEARCH OBJECTIVES.2 

D. SCOPE AND DELIVERABLES.4 

E. POTENTIAL BENEFITS.5 

F. LIMITATIONS.6 

G. METHODOLOGY.8 

H. ORGANIZATION OF REPORT.10 

I. SUMMARY.11 

II. LITERATURE REVIEW.15 

A. INTRODUCTION.15 

B. CONTINGENCY CONTRACTING.16 

1. History of Contingency Contracting Pre-9/11.16 

2. History of Contingency Contracting Post-9/11.18 

3. Problems and Challenges in Contingency Contracting.19 

4. Formalization of Contingency Contracting Officer Training 

and Tasks.21 

5. Other Established Training Methods.27 

6. Defense Acquisition University CON 234, Contingency 

Contracting Course.28 

7. Contingency Contracting Officer Handbook.29 

8. Soldier’s Manual of Common Tasks.29 

C. TESTING AND EVALUATION.30 

1. Training and Assessment.30 

2. Core Test Principles.38 

3. Test Construction.42 

4. Test Blueprints.43 

5. Item W riting.46 

D. SUMMARY.48 

III. CONTINGENCY CONTRACTING FRAMEWORK—THE ECC AND THE 

SMCT .51 

A. INTRODUCTION.51 

B. ESTABLISHMENT OF THE ECC.51 

C. ECC ORGANIZATIONAL STRUCTURE.53 

D. TRAINING METHODOLOGY.53 

E. CONTRACT MANAGEMENT OVERVIEW.56 

F. SMCT.57 

G. SUMMARY.59 

IV. PAT DEVELOPMENT.61 

A. INTRODUCTION.61 

vii 









































B. TEST CONSTRUCTION PLAN.61 

C. TEST BLUEPRINT.61 

D. ITEM WRITING.62 

E. VALIDITY AND RELIABILITY.62 

F. COGNITIVE DOMAINS.64 

G. SUMMARY.65 

V. SUMMARY, CONCLUSION, AND AREAS FOR FURTHER RESEARCH.67 

A. SUMMARY.67 

B. CONCLUSION.68 

C. AREAS FOR FURTHER RESEARCH.70 

LIST OF REFERENCES.73 

INITIAL DISTRIBUTION LIST.77 


viii 















LIST OF FIGURES 


Figure 1. Army Training and Leader Development Model.31 

Figure 2. Theoretical Foundation of Adult Learning.32 

Figure 3. Army Training Planning Process (Training Assessment).33 

Figure 4. Core Adult Learning.36 

Figure 5. Revised Bloom’s Taxonomy of Learning Objectives.45 

Figure 6. Revised Bloom’s Taxonomy of Learning Objectives.46 

Figure 7. Example of Selected-response Item and Test-item Nomenclature.47 

Figure 8. ACC and ECC Organizational Structure.52 

Figure 9. CCO Development Model.54 

Figure 10. CCO Development Timeline.56 

Figure 11. 36 MOS 51 Common Tasks ( SMCT ).58 














THIS PAGE INTENTIONALLY LEFT BLANK 


x 



LIST OF ACRONYMS AND ABBREVIATIONS 


ACA 

Army Contracting Agency 

ACC 

Army Contracting Command 

AFARS 

Army Federal Acquisition Regulation Supplement 

AFIT 

Armed Forces Institute of Technology 

AR 

Army Regulation 

AT&L 

Acquisition Technology and Fogistics 

CCBn 

Contingency Contracting Battalion 

CCE 

Contracting Center of Excellence 

CCO 

Contingency Contracting Officer 

CON234 

Contingency Contracting Course 

CONUS 

Continental United States 

CSB 

Contingency Support Brigade 

CWC 

Commission on Wartime Contracting 

DAU 

Defense Acquisition University 

DAWIA 

Defense Acquisition Workforce Improvement Act 

DFARS 

Defense Federal Acquisition Regulation Supplement 

DoA 

Department of the Army 

DoD 

Department of Defense 

DTIC 

Defense Technical Information Center 

ECC 

Expeditionary Contracting Command 

FAM 

Functional Area Managers 

FAR 

Federal Acquisition Regulation 

FM 

Field Manual 


xi 



GAO 

Government Accountability Office 

LTC 

Fieutenant Colonel 

MAJCOM 

Major Command 

MOS 

Military Occupational Specialty 

NPS 

Naval Postgraduate School 

ODS 

Operation Desert Storm 

OEF 

Operation Enduring Freedom 

OIF 

Operation Iraqi Freedom 

OUSD 

Office of the Under Secretary of Defense 

PAT 

Proficiency Assessment Test 

SF 

Standard Form 

SMCT 

Soldier’s Manual of Common Tasks 

USAID 

United States Agency for International Development 

USMC 

United States Marine Corps 


xii 



ACKNOWLEDGMENTS 


The members of the research team collectively wish to thank our academic 
advisors at the Naval Postgraduate School. In particular, we wish to thank Jeffrey 
Cuskey, CDR, USN (Retired) who introduced us to the project but who was unable to see 
its completion. We would also like to thank Professor Rene Rendon, PhD., LtCol, USAF 
(Retired) for his mentorship throughout our tenure at NPS, and Professor Cory Yoder, 
CDR, USN (Retired) for his contribution. Without their support and mentorship, this 
project would not have been possible. 

Additionally, we would like to thank key members of the Expeditionary 
Contracting Command who assisted and guided us throughout the project. Most notable 
among them were Lieutenant Colonel Tim Strange and Brigadier General Joe Bass. 

Individually, we wish to thank our families for their tireless support and patience 
while we worked on this project. Most important of all, we give thanks to God for the 
multitude of blessings in our lives. 



THIS PAGE INTENTIONALLY LEFT BLANK 


xiv 



EXECUTIVE SUMMARY 


U.S. Army contingency contracting officers (CCOs) play a critical role in nearly all 
overseas military operations by providing supplies and services to the warfighter in an 
efficient, timely, and ethical manner. Despite this critical role, the Army has not 
integrated its CCOs or the procurement process in a holistic approach to its 
expeditionary, warfighting mindset. Shortfalls in the number of U.S. Army CCO 
personnel and the lack of a standardized training process have negatively affected the 
Army’s expeditionary procurement capabilities. 

The U.S. Army’s Expeditionary Contracting Command (ECC) was established in 
January 2008 in direct response to the Gansler Commission’s recommendations to 
prepare, train, and provide skilled contracting personnel in support of U.S. Army 
contingency operations. This new command, and other resultant changes of the Gansler 
Commission’s recommendations, resulted in a planned net increase of over 400 
uniformed and 800 civilian contracting officer positions. 

The ECC developed a standardized set of 36 tasks to properly train CCOs prior to 
deployment. These 36 tasks, published in the ECC’s Soldier’s Manual of Common Tasks 
(SMCT) MOS 51C Acquisition, Logistics, and Technology, represent the body of 
knowledge that a 51C CCO must know to be successful during contingency operations. 

The establishment and standardization of training tasks by the ECC for CCOs in 
the MOS 51C SMCT, in accordance with Army training doctrine, has helped subordinate 
units and individual CCOs within the ECC to focus their training efforts prior to 
deployment. This body of knowledge is a significant step in standardizing the pre¬ 
deployment training for CCOs and preparing them for their mission once deployed. To 
supplement this training, an appropriate performance-assessment tool is needed to 
measure performance of these 36 tasks. Currently, the ECC lacks a standardized method 
of measuring individual skill and task proficiency in its CCOs. 
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A written proficiency-assessment test will allow leaders in the ECC to measure a 
CCO’s knowledge, skills, and capabilities. A written PAT will enable leaders to measure 
individual task proficiency against established standards, determine each CCO’s 
preparedness to conduct duties during deployment, make appropriate CCO assignments, 
and develop individualized training programs for CCOs. 

This research developed the questions for a written proficiency test based on the 
36 tasks listed in the SMCT. The research team developed the scenarios and questions to 
assess proficiency in accordance with conditions, standards, and performance steps for 
each of the 36 tasks in the SMCT. Questions were structured to assess Bloom’s cognitive 
domains of remembering, understanding, and, to a limited extent, application of tasks. 
The research team vetted each of the scenarios and accompanying questions through 
experienced CCO focus groups within the ECC. Upon completion of the test, the 
research team provided a written Contracting Officer Proficiency-Assessment Test, 
encompassing over 1,100 questions covering each of the 36 tasks listed in the SMCT , to 
the Expeditionary Contracting Command. 



I. 


INTRODUCTION 


A. BACKGROUND 

U.S. Army contingency contracting officers (CCOs) play a critical role in nearly 
all overseas military operations by providing supplies and services to the warfighter in an 
efficient, timely, and ethical manner. Despite this critical role, the Army has not 
integrated its CCOs or the procurement process in a holistic approach to its 
expeditionary, warfighting mindset. Shortfalls in the number of U.S. Army CCO 
personnel and the lack of a standardized training process have negatively affected the 
Army’s expeditionary procurement capabilities (Gansler, 2007). 

The 21st century has brought about significant changes in the manner in which 
U.S. Forces conduct operations overseas. The post-Cold War reduction in forces saw a 
decrease in acquisition workforce professionals throughout the Army. At the same time, 
the United States military, and in particular the Army, significantly increased the 
outsourcing of many jobs that were previously performed by uniformed Service members 
(Gansler, 2007). This decrease in contracting professionals and increase in contracting 
requirements led to many deficiencies highlighted in the Gansler Commission report 
(2007). 

The U.S. Army’s Expeditionary Contracting Command (ECC) was established in 
January 2008 in direct response to the Gansler Commission’s recommendations to 
prepare, train, and provide skilled contracting personnel in support of U.S. Army 
contingency operations (Gansler, 2007). This new command, and other resultant changes 
of the Gansler Commission’s recommendations, resulted in a planned net increase of over 
400 uniformed and 800 civilian contracting officer positions (DoA, 2009a). Most of the 
new uniformed positions within the ECC are coded “51C,” the U.S. Army’s Military 
Occupational Specialty (MOS) code for contracting officer. The ECC developed a 
standardized set of tasks to properly train CCOs prior to deployment. These 36 tasks, 
published in the ECC’s Soldier’s Manual of Common Tasks (SMCT) MOS 51C 
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Acquisition, Logistics, and Technology , represent the body of knowledge that a 51C CCO 
must know to be successful during contingency operations (DoA, 2009a). 

This research builds upon the 36 tasks listed in MOS 51C SMCT by building a 
proficiency-assessment test (PAT) for the SMCT tasks. The PAT developed throughout 
this research is nested with Army training doctrine prescribed in U.S. Army Field Manual 
(FM) 7-0 and provides an assessment and evaluation tool for individual task proficiency 
in CCOs. This assessment “provide[s] the link between the unit’s [or individual’s] 
performance and the [established] Army standard,” as prescribed in the SMCT (DoA, 
2002 ). 

B. PROBLEM STATEMENT 

The establishment and standardization of training tasks by the ECC for CCOs in 
the MOS 51C SMCT , in accordance with Army training doctrine, has helped subordinate 
units and individual CCOs within the ECC to focus their training efforts prior to 
deployment. This body of knowledge is a significant step in standardizing the pre¬ 
deployment training for CCOs and preparing them for their mission once deployed. To 
supplement this training, an appropriate performance-assessment tool is needed to 
measure performance of these 36 tasks. Currently, the ECC lacks a standardized method 
of measuring individual skill and task proficiency in its CCOs. The ECC determines 
CCO duty assignments and warrant levels without a quantifiable measure of 51C 
individual task proficiency. A written proficiency-assessment test will allow leaders in 
the ECC to measure a CCO’s knowledge, skills, and capabilities. A written PAT will 
enable leaders to measure individual task proficiency against established standards, 
determine each CCO’s preparedness to conduct duties during deployment, make 
appropriate CCO assignments, and develop individualized training programs for CCOs. 

C. RESEARCH OBJECTIVES 

The objective of this research is to develop a written proficiency test for the 36 

tasks listed in the SMCT. The research team developed the scenarios and questions to 

assess proficiency in accordance with conditions, standards, and performance steps for 

each of the 36 tasks in the SMCT. Questions were structured to assess Bloom’s cognitive 
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domains of remembering, understanding, and, to a limited extent, application of tasks 
(Bloom, 1956). The research team vetted each of the scenarios and accompanying 
questions through experienced CCO focus groups within the ECC. Upon completion of 
the test, we provided a written Contracting Officer Proficiency-Assessment Test, 
covering each of the 36 tasks listed in the SMCT , to the Expeditionary Contracting 
Command. This final document included written documentation of scenarios, questions, 
and answers developed by us. 

To meet this objective, this research answered the following primary research 
question: 

What is the most effective and efficient method of measuring Individual CCO 
task proficiency in the 36 tasks listed in the SMCT ? 

Secondary research questions were as follows: 

• How can a proficiency-assessment test be developed? 

• What type(s) of testing is best suited to evaluate the cognitive domains of 
remembering, understanding, and, to a limited extent, application of 
knowledge? 

• What type(s) of testing is most efficient—in terms of requiring the least 
amount of resources, such as overhead and setup—for the tester and test 
taker? 

This research was completed when scenario(s) and accompanying questions for 
each of the 36 tasks had been developed, the scenarios and questions were vetted through 
focus groups of the ECC, and a final written version of the PAT was submitted and 
received by the ECC. Additionally, the research team provided the ECC with 
recommendations as to the most effective format in which to issue the test. 


3 



D. SCOPE AND DELIVERABLES 


The scope of this research project was limited to developing a test to assess 
proficiency in the 36 tasks listed in the SMCT. The specific deliverables at the 
conclusion of this research project were specifically designed to satisfy the research 
project objectives. The research team accomplished the following: 

1. Developed a written proficiency-assessment test (PAT) that measures proficiency 
in each of the 36 tasks listed in the SMCT. The PAT included the following: 

a. Written scenarios for each of the 36 tasks. 

b. Questions for each of the scenarios that measure the cognitive domains of 
remembering, understanding, and, to a limited extent, application of 
knowledge of the 36 tasks. 

2. Developed the scenarios and accompanying questions to assess proficiency in 
accordance with the conditions, standards, and performance steps listed in the 
SMCT. 

3. Vetted each scenario and the accompanying questions through focus groups 
comprised of experienced CCOs and members of the ECC to ensure the scenario 
and questions satisfy the conditions, standards, and performance steps listed in the 
SMCT. 

4. Built the PAT in a manner that allows it to be converted into a computerized 
assessment test. 

5. Provided a written version of scenarios, questions, and answers to the ECC upon 
completion. 

The ECC has established the 36 tasks in the SMCT as the body of knowledge that 
a CCO must know to be successful. The research team accepted these tasks and focused 
on developing the test to satisfy proficiency in that body of knowledge. Analysis of the 
relevance or validity of the 36 tasks as they relate to current, changing, or emerging 
trends in CCO tasks were not within the scope of this project. As such, this research did 
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not analyze the validity of tasks listed in the SMCT. However, validation of tasks could 
be a subject area of further research interest by follow-on researchers. 

E. POTENTIAL BENEFITS 

A written proficiency-assessment test will provide essential feedback to Army 
contracting leaders by enabling them to measure individual task proficiency against 
established standards. Leaders could administer the PAT to newly arrived CCOs to 
gauge what tasks they are proficient in and what tasks they need work in. This 
knowledge will allow CCOs and their leaders to develop specifically tailored and 
individualized training programs to address weaknesses in the CCOs knowledge base, as 
demonstrated in the PAT results. Follow-on PATs can help the leader and CCO see 
progress in the individual training program. Using the PAT as the training feedback 
mechanism will allow leaders and individuals to determine if their training focus and 
methodology is accurate or should be adjusted. Moreover, the PAT will serve as valuable 
feedback to a commander on his unit’s training program. 

Leaders in the ECC will be able to measure a CCO’s knowledge, skills, and 
capabilities and determine a CCO’s preparedness to conduct duties during deployment. 
The PAT can serve as a capstone certification of an individual’s deployment readiness at 
the conclusion of an extensive training program and prior to a scheduled deployment. 
The successful passing of the PAT will give the individual CCO confidence in his 
abilities to perform his mission. Moreover, it will serve as a validation to the individual’s 
leaders that the CCO is ready to accomplish his mission. 

A PAT may help ECC leaders assign CCOs to positions that match the CCO’s 
experience and skill qualifications. The ECC has a variety of positions available for 
assignment, some coded for more experienced CCOs and some for entry-level CCOs. 
Two classic examples of positions are the CCO team member and CCO team leader. The 
first-line supervisors make detailed assignment adjustments at the unit level once a CCO 
has arrived and has been assessed over time. A PAT would allow ECC leaders and first 
line supervisors to distinguish between newly arrived or inexperienced CCOs and ones 
with more experience. 
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F. 


LIMITATIONS 


The PAT, as developed, has several limitations. The limitations of the final 
version of the PAT and its accompanying answers include susceptibility to compromise 
by would-be test takers, a limited ability to adapt to emerging trends in contingency 
contracting, inability to accurately reproduce the interface and interaction required of 
CCOs, and inability to measure experience accurately. Leaders within the ECC can issue 
the PAT with effective results if they understand these limitations and make 
accommodations beforehand to account for them. These limitations should be included 
in the test instructions by the test administrators. 

The static nature of the final PAT as a written test makes it susceptible to 
compromise via memorization or copying and dissemination. While outside the scope of 
the research project, the final version of the test should be a computerized version of the 
PAT that can be easily transported, administered, taken, graded, and evaluated. The 
precursor to such a digitized version is the written PAT presented in this research project. 
This product has written scenarios and accompanying questions that satisfy the task, 
conditions, and performance steps in the SMCT. The digitized version will have to 
present these questions to each scenario in a random manner. This random presentation 
will limit test memorization and compromise and, by extension, its exploitation by 
would-be test takers. 

The PAT developed in this research is designed to test proficiency in the 36 tasks 
listed in the SMCT. The SMCT signifies the fixed body of knowledge that the ECC has 
determined that a CCO must master to become successful as a CCO. This PAT is 
developed as a measure of proficiency of that body of knowledge and is, therefore, not 
easily adaptable to emerging trends in contingency contracting. In researching the 
previously trained on CCO tasks, we found a varying degree of recommended tasks for 
CCO training over the past 15 years, many of which are not consistent with the 36 tasks 
in the SMCT (Tigges & Snyder, 1993; Lasch, 2002; Kirstein, 2003). As a result, we 
make the assumption that as the nature of warfare changes, so too will the nature of 
contingency contracting change. It is possible that the tasks specified in the SMCT may 

change, over time, with what CCOs face as the nature of warfare, regulations, and 
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deployments change. The fixed tasks in the PAT may become outdated and need 
revision. It is not within the scope of this project to evaluate or validate the 36 tasks 
listed in the SMCT. ECC leaders, PAT administrators, and CCOs should keep in mind 
the dynamic nature of contingency operations and their contracting needs. The research 
team recommends that the PAT become an evolving instrument, the content of which is 
managed in some form by the ECC. After its inclusion as a training tool for the ECC, 
obvious or glaring omissions not tested should be identified by the ECC for 
incorporation, and extraneous tasks tested should be identified for exclusion in future 
editions of the PAT. Periodic reviews and doctrinal revisions of the SMCT should 
similarly trigger PAT revisions. In the interim, ECC leaders and CCOs should be aware 
of emerging trends and develop solutions to test for tasks not listed, or omit tasks from 
the test that are no longer relevant. 

Contingency contracting is a skill that takes years to develop. An individual’s 
development consists not just of completing educational requirements but also of gaining 
experience to make decisions in a dynamic and complex environment. Contingency 
contracting is as much an art as it is a science. It is making the most appropriate 
decisions among a multitude of scenarios and variables that makes for a well-trained 
CCO. Whether in written format or as a computerized version, no form of test, unless it 
is performance-based, can accurately capture and measure the “art” of contingency 
contracting, developed over years of experience. No written or computerized test can 
reproduce the interface and interaction required to accomplish many of the 36 SMCT 
tasks. The PAT is limited in this respect. The research team recognizes that it cannot 
fully measure a CCO’s level of proficiency. However, the PAT can be developed to 
measure a baseline of knowledge in the 36 tasks by presenting scenarios and questions 
from which the CCO must choose the right answer. Additionally, the PAT was 
developed by the research team to measure a certain level of proficiency by creating 
scenarios and accompanying questions that will trigger experienced CCOs to search for 
the “best answer.” 
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G. 


METHODOLOGY 


The research team began the research by conducting a literature review of all 
pertinent contingency-contracting literature. Contingency contracting is a niche subject 
area in the greater realm of contract management, executed almost exclusively by the 
Department of Defense (DoD). As such, we found very few published articles in 
established scholarly journals, practitioner journals, general-interest magazines, or other 
publications. Moreover, the specific nature of the project, to develop a test for CCOs, 
further narrowed the scope of applicable journals. The databases of the Defense 
Technical Information Center (DTIC) and Inside Defense provided the majority of 
articles germane to the research topic. The research team reviewed all pertinent and 
current articles covering contingency contracting to provide context in the development 
of the test. 

The literature review chapter covers the history of contingency contracting before, 
and its evolution since, Operation Desert Storm (ODS). This evolution encompasses the 
drawdown of much of the Army’s acquisition workforce during the early 1990s and the 
near simultaneous increase in operational requirements during that time following the 
terrorist attacks of September 11, 2001. Furthermore, the literature review covers the 
resultant fallout of these changes, culminating in the Gansler Commission findings and 
recommendations. By understanding contingency contracting from a historical 
perspective, the research team established a context for developing the PAT. 

Critical to the literature review was the full understanding of the ECC as an 
organization. As the research sponsor and customer, the ECC required a final, usable 
PAT that would support their needs. This includes understanding the ECC’s role and 
purpose, command structure, subunits, training methodology, and where in its growth-to- 
completion the ECC stands. The ECC was created in direct response to the Gansler 
Commission’s recommendations to prepare, train, and provide skilled contracting 
personnel in support of U.S. Army contingency operations (Gansler, 2007). It is a newly 
created Army organization under command of a one-star general. Figure 8 shows the 
ECC’s organization under the Army’s Contracting Command. At the time of this 

writing, the ECC, as a new and large organization, is in the midst of growing and has yet 
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to reach its final end-strength. It was important for us to understand where the ECC 
stood in terms of its development as an organization in order to understand how the PAT 
complemented the ECC’s training focus and methodology and how the PAT could 
remain relevant and useful over time. 

The SMCT represents the cornerstone of this research project since this single 
document represents the body of knowledge upon which the PAT was developed. The 
SMCT lists the 36 tasks and performance steps to successfully accomplish each task. The 
performance steps are, in essence, the correct answer for each accompanying task. The 
SMCT also articulates what role, in the greater scheme of Army training doctrine, the 
SMCT fills. In developing the PAT, it was critical for us to understand the tasks within 
the SMCT , how their proficiency is measured in the SMCT , and how this overall 
document fits in the ECC’s training methodology. 

Additionally, no official test to augment a U.S. Army unit’s training methodology 
would be complete without a thorough review of Army training doctrine. The research 
team reviewed US Army Field Manuals (FM) 7-0(Training the Force) (2002), 7-l{Battle 
Focused Training) (2003), and Army Regulation (AR) 350-l(Army Training and Leader 
Development). Collectively, these three documents represent the cornerstone of U.S. 
Army training doctrine and methodology, and we used them to develop the PAT in 
context as a formal supplement to the SMCT. 

Lastly, the research team conducted a review of testing and assessment theories. 
This review included theoretical application of assessment as a subset of the adult¬ 
learning model, core test principles, test construction and test blueprints, Bloom’s 
Taxonomy , and item writing as it relates to test construction. The knowledge we gained 
in reviewing test construction and evaluation-practices literature allowed us to choose the 
best practices to develop a valid and reliable proficiency-assessment test in a time- 
constrained environment that meets the needs of the Expeditionary Contracting 
Command and U.S. Army CCOs. 
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In developing the PAT, the research team took each task and broke it down to its 
subcomponents of task, conditions, and performance steps. The tasks in the SMCT 
represent the 36 testable tasks of PAT that we used to measure proficiency. The 
conditions for each task “identify all the equipment, tools, references, job aids, and 
supporting personnel that the Soldier needs to use to perform the task in wartime” (DoA, 
2009, September 22, p. 1-3). The performance steps in the SMCT are essentially the 
correct way to demonstrate proficiency. They are, in effect, the answers to the test. By 
using the task, conditions, and performance steps in the SMCT , the research team reverse- 
engineered each of the 36 tasks in the SMCT to create scenarios and questions for the 
answers of the test. 

Once we developed a scenario or group of scenarios and their accompanying 
questions and answers, we sent the scenarios electronically to the ECC for vetting. The 
research team’s sponsor and point of contact at the ECC distributed each of the scenarios 
to varying groups of experienced and senior leaders within the ECC. The research team’s 
sponsor and the group of vetting officers are all experienced CCOs. This group 
developed the SMCT for the MOS 51C. Additionally, they vetted the scenarios to ensure 
that they satisfied the requirements for each of the tasks listed in the SMCT. If any of the 
tasks needed revision, the vetting group submitted any recommended changes to our 
sponsor, who, in turn, submitted them to the research team. The research team had no 
direct contact or correspondence with the vetting group of officers. All scenario 
submissions, changes, and revisions were handled through our sponsor on behalf of the 
research team. Upon completion of all 36 tasks, the research team compiled the 36 
scenarios with accompanying questions and answers on a single Word document. While 
not a part of the scope of this project, the research team will continue to work towards 
converting the written version of the PAT to a computerized assessment test. 

H. ORGANIZATION OF REPORT 

This report is organized in the following manner. Chapter I, Introduction, 
provides information on the background of this research project, lists the problem 
statement and research objectives, defines the scope and deliverables of the project, lists 
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potential benefits and limitations of the project, and describes the methodology by which 
the research was conducted. Chapter II, Literature Review, covers all pertinent articles 
from scholarly journals, practitioner journals, general-interest magazines, and other 
publications on the topics of training and evaluating contingency contracting and on test 
construction and knowledge assessment. Chapter III, Contingency-Contracting 
Framework, looks in detail at the organization responsible for the SMCT, the 
Expeditionary Contracting Command, and covers its history, organizational structure, and 
training focus. Additionally, this chapter describes in detail the SMCT , the document that 
represents the body of knowledge for the PAT. Chapter IV, PAT Development, describes 
the steps the research team took in developing the PAT. Lastly, Chapter V, Conclusion, 
summarizes the research project and makes recommendations for areas of further 
research. 

I. SUMMARY 

U.S. Army CCOs have been and continue to be a critical force multiplier in any 
contingency operation. Changes in the force structure of U.S. Army acquisition, coupled 
with increasing demands on CCOs due to the changing nature of warfare, led to a 
shortage of a qualified and capable CCO base for the U.S. Army. This shortage caused 
the Army to rely heavily on sister Services to provide CCOs (Gansler, 2007). In response 
to the Gansler Commission findings, and in an effort to better support the warfighter, the 
U.S. Army established the Expeditionary Contracting Command in Fort Belvoir, 
Virginia. This new command, under a one-star general leadership, is tasked with 
“providing skilled, trained, contracting personnel for the support of expeditionary forces” 
(Gansler, 2007, p. 52). 

This new command, and other resultant changes of the Gansler Commission’s 
recommendations, resulted in a planned net increase of over 400 uniformed contracting 
officer positions (DoA, 2009a). Many of these positions were for CCOs, many of whom 
have no previous training in their new field of contracting, MOS 51C. To standardize 
training across the ECC for all CCOs, the ECC developed the MOS 51C SMCT as the 
body of knowledge needed by CCOs to be successful contracting officers during 
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deployment. While this was a significant step in formalizing the ECC’s training 
methodology, no method of assessing performance of these tasks existed. 

The research team, in partnering with the ECC, developed the Contingency 
Contracting Officer Proficiency-assessment Test (PAT). The PAT is based on the 36 
tasks listed in the SMCT. The PAT consists of a scenario or scenarios for each of the 36 
tasks. Each scenario has accompanying questions that measure Bloom’s cognitive 
domains of remembering, understanding, and, to a limited extent, application of 
knowledge of the tasks as defined by the performance steps. Proficiency of each task is 
demonstrated by successfully answering questions that meet the criteria in the 
performance steps. 

The written version of the PAT has several limitations. As a static, written test, it 
is vulnerable to compromise and dissemination. Moreover, because it is based on the 36 
tasks in the SMCT , it is not easily adaptable to emerging trends in contingency 
contracting. Lastly, a written test can never fully capture the “art” of contingency 
contracting. The learning curve for a CCO is too long and the variables too diverse to 
fully capture the nuance of possible choices to a “right” answer. Despite these 
limitations, a PAT will serve a critical role in the Army and ECC training methodology 
by providing the feedback mechanism to CCOs, ECC leaders, commanders, and trainers 
responsible for the professional development of CCOs. 

The PAT developed in this research project will serve to augment the ECC’s 
training methodologies by providing a critical need in validating task proficiency in the 
36 tasks listed in the SMCT. While not within the scope of this project, a digitized 
version of the PAT will make the PAT easily transferable and accessible. The products 
of this research project will provide a critical feedback to ECC CCOs and their leaders by 
providing feedback on training and experience. The PAT will not only help the ECC 
continue to grow as an organization but also help it grow its newly assessed CCOs into 
competent and capable CCOs who serve as force multipliers for U.S. Army warfighters 
in current and future contingencies. 
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The next chapter, Literature Review, will cover all pertinent articles from 
scholarly journals, practitioner journals, general-interest magazines, and other 
publications on the topics of training and evaluating contingency contracting and on test 
construction and knowledge assessment. With respect to contingency contracting, the 
literature review focuses on the history of contingency contracting and previous training 
tasks to train CCOs. In the realm of test construction, the research team focused on the 
theoretical and academic study of the adult-leaming model, core test principles, the 
practical application of test blueprints, item writing, and test construction, all within the 
construct of Bloom’s Taxonomy. 
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II. LITERATURE REVIEW 


A. INTRODUCTION 

Developing a proficiency-assessment test for contingency contracting officers 
brings together two highly specialized and disparate fields: contingency contracting and 
test construction and evaluation. Contingency contracting is a niche subject area within 
the greater realm of defense contracting. This literature review focuses on the history of 
contingency contracting, with an emphasis primarily on training and evaluation of CCOs. 
The research team wanted to understand trends and changes in CCO training and 
evaluation throughout modern military history in order to help build an effective 
proficiency-assessment test. We understood that to develop a proficiency-assessment test 
would require extensive knowledge of test construction theory and practices. This 
research focuses on the theoretical and academic study of the adult-learning model, core 
test principles, practical application of test blueprints, item writing, and test construction 
within the construct of Bloom’s Taxonomy. The research team used this knowledge to 
create the most effective and efficient proficiency-assessment test for the 36 tasks in the 
SMCT, given the limited time and expertise constraints of the research team. 

A thorough review of related literature revealed an abundance of research efforts 
and published articles on contingency-contracting-related topics. However, this literature 
review concentrates on the evolution of U.S. Army contingency contracting as it relates 
to training, assessing, and preparing CCOs for deployment to hostile environments, a 
narrower field of study with less available research literature. Areas covered include the 
following: the history of contingency contracting, the problems and challenges of 
contingency contracting, the history of established tasks related to contingency 
contracting, and the evolution of contingency contracting since 9/11. By understanding 
contingency contracting from a historical perspective, the research team established a 
context for developing the PAT. Moreover, by reviewing the historical training tasks for 
CCOs, the research team gained an understanding of the SMCT as the base document for 
the development of the PAT. 
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B. CONTINGENCY CONTRACTING 


U.S. Army contingency contracting is a highly functional and narrowly focused 
area of study. As a result, there is little relevant literature to be extracted from civilian 
scholarly journals, practitioner journals, general-interest magazines, or other publications. 
The research team reviewed the National Contract Management Association’s database 
for relevant articles. Comparable studies that specifically addressed the issue of 
contingency-contracting training were very limited. The research team reviewed 
databases of the Office of the Inspector General with similar results. The databases of the 
Defense Technical Information Center (DTIC) and Inside Defense proved to be the best 
research databases to search for contingency-contracting-related topics. A search of 
those two databases provided several contingency-contracting-related studies conducted 
by graduate students at the Naval Postgraduate School (NPS) and the Air Force Institute 
of Technology (AFIT). These studies—along with Government Accountability Office 
(GAO) reports, the Gansler Commission report, and DoD and Air Force guidance—are 
the foundation of this portion of the literature review and study. 

1. History of Contingency Contracting Pre-9/11 

Contingency-contracting officers have played an important role throughout 
modem military. The origins of contingency contracting in the U.S. military can be 
traced back to the American Civil War (Kirstein, 2003). Mason (1988) provided a 
timeline outlining the role of procurement personnel from that time period to 2003. 
Contracting officers of the 1860s, then called quartermasters, were directly assigned to 
army divisions fighting the Civil War. As an integral part of the force, the Quartermaster 
corps reduced the need for a logistical footprint to support the war effort (Mason, 1988). 

During World War I and World War II, contingency contracting personnel played 
a lesser role because most supplies were shipped from the United States (Kirstein, 2003). 
Shipping supplies from the United States by watercraft proved both slow and inflexible to 
the needs of the warfighter. Goods often arrived long after the need for them had passed. 
This resulted in surpluses in some items and severe shortages in others. On a limited 
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scale, the British carried out contracting tasks by providing surplus goods to supplement 
those arriving from the United States (Mason, 1988). 

During the Korean War, contracting was used almost exclusively for providing all 
classes of supplies for the troops, marking the first time the troops relied on contracting 
as an important procurement vehicle. The ability of local vendors to provide basic items 
reduced the need for shipments from the United States. Japanese and Korean contractors 
provided extensive support to the war effort. Local procurement of supplies resulted in a 
flexible supply chain that was responsive to the warfighter’s needs (Kirstein, 2003). 
During the Vietnam conflict, the utilization of contracts to procure supplies and services 
greatly declined. Mason attributes this decline to the fact that a formal war declaration 
was never enacted by the United States Congress. Without a formal declaration of war, 
the Air Force did not have the political support or resources it needed to provide optimal 
support (Mason, 1988). 

During Operation Desert Storm and Desert Shield (ODS), contingency 
contracting as we know it today was still maturing. Contingency-contracting personnel 
were able to provide unprecedented support to the warfighters (Kirstein, 2003), but faced 
many challenges as they attempted to procure supplies and services. Their biggest 
challenge came from the Government’s own regulations (Luse, Madeline, Smith & Starr, 
2005). Restrictions imposed by the Federal Acquisition Regulation (FAR), Defense 
Federal Acquisition Regulation Supplement (DFARS), and Army Federcd Acquisition 
Regulation Supplement (AFARS) made contingency-contracting operations more difficult 
than they needed to be. These regulations were not designed for contingency operations, 
and their applicability under wartime conditions was vague (Luse, 2005). In his book 
Moving Mountains: Lessons in Leadership and Logistics in the Gulf War, General 
Pagonis (1992) described the daunting mission of contracting personnel, stating “our 
limited and precious transport space should be reserved for combat troops, and for these 
supplies, such as weapons and ammunition [...]. Everything else was our problem to be 
found and contracted for” (Pagonis, 1992, p. 107). During ODS, most logistical needs 
were met, at least in part, by CCOs. Items and services contracted for included food, 
water, lodging, laundry, sanitation, communications, transportation, and miscellaneous 
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equipment (Pagonis, 1992). Both the Army and the Air Force relied heavily on contract 
leases and procured goods and services. The lessons learned from Operation Desert 
Storm shaped current CCO practices and are being applied to meet the challenges of 
Operation Iraqi Freedom and Operation Enduring Freedom (Kirstein, 2003). 

2. History of Contingency Contracting Post-9/11 

The 21st century has brought about significant changes in the manner in which 
U.S. Forces conduct operations overseas. The post-Cold War reduction in forces saw a 
decrease in acquisition workforce professionals throughout the Army. From 1990 to 
1996, the U.S. Army saw its acquisition workforce decrease from approximately 10,000 
personnel to approximately 5,500 personnel, where it remained through 2001 (Gansler, 
2007). At the same time, the United States military, and in particular the Army, 
significantly increased the outsourcing of many jobs that were previously performed by 
uniformed Service members. As an indicator of complexity in contracting, the value of 
U.S. Army contracts increased 331% from $23.3 billion in 1992 to $100.6 billion in 
2006, and the number of contracts actions, as an indicator of workload, increased 654% 
from approximately 52,900 to approximately 398,000 over the same timeframe (Gansler, 
2007). This decrease in contracting professionals and increase in contracting 
requirements led to many deficiencies in the Army’s ability to conduct procurements for 
its warfighters during contingency operations. 

Operation Iraqi Freedom (OIF) and Operation Enduring Freedom (OEF) have 
seen the largest use of contingency contracting by the Department of Defense in its 
history. Operations in Iraq and Afghanistan have demonstrated the demand that 
expeditionary military operations place on the contracting system and contracting 
personnel. A 2008 GAO report analyzed data from the DoD, Department of State, and 
United States Agency for International Development (USAID) on contracting activities in 
Iraq and Afghanistan for fiscal year (FY) 2007 and the first half of FY 2008. This report 
reviewed (1) the number and value of contracts and the extent they were awarded 
competitively, (2) the number of contractor personnel, including those performing 
security functions, and (3) the number of contractor personnel who were killed or 
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wounded. The report showed that the United States had more civilian contractor 
personnel in Iraq in 2008 than it did U.S. Army soldiers. These contractor employees 
provide food services, interpretation services, communications, equipment repair, and 
other important services. 

3. Problems and Challenges in Contingency Contracting 

The shortfall in trained U.S. Army contracting personnel in Iraq and Afghanistan 
and the exponential increase in contingency-contracting procurement requirements led to 
failures highlighted in the 2007 report by the Gansler Commission. Officially called 
Urgent Reform Required: Army Expeditionary Contracting, the Commission on Army 
Acquisition and Program Management in Expeditionary Operations (led by Dr. Jacques 
S. Gansler, former Under Secretary of Defense (Acquisition, Technology & Logistics)) 
examined the Army’s business processes as they relate to expeditionary contracting 
operations and highlighted the shortfalls and problems experienced by the Army prior to 
October 2007. The report found that some critical segments of the “Institutional Army” 
had not adapted to the expeditionary acquisition requirements, listing these “key failures” 
as “significantly contributing to the fraud, waste, and abuse in-theater” by Army 
personnel (Gansler, 2007, p. 1). Of these critical segments, two, training and education 
and doctrine, regulations and processes, are addressed in some measure by our 
development of the proficiency-assessment test. The PAT is nested in newly developed 
doctrine, as found in the SMCT. Moreover, it provides a critical feedback mechanism for 
CCOs and their commanders on CCO training plans. 

The Gansler Commission recommended four overarching changes for the Army: 

1. Increase the stature, quantity, and career development of 
military and civilian contracting personnel (especially for 
expeditionary operations). 

2. Restructure organization [Army] and restore responsibility to 
facilitate contracting and contract management in expeditionary 
and CONUS [Continental United States-based] operations. 

3. Provide training and tools for overall contracting activities in 
expeditionary operations. 
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4. Obtain legislative, regulatory, and policy assistance to enable 
contracting effectiveness in expeditionary operations. (Gansler, 

2007, p. 5) 

The establishment of the Expeditionary Contracting Command is a direct result of 
recommendation number two above. The ECC’s development of the SMCT as a standard 
of knowledge for CCOs is in direct response to the Gansler Commission’s third 
recommendation. In the Gansler report, the Commission further elaborates on their third 
recommendation by adding “Develop and field contract ‘tools’ needed for expeditionary 
forces including, but not limited to, sample contracts, statements of work, pre-agreed-to 
waivers, etc.” (Gansler, 2007, p. 55). It is this recommendation that necessitates the 
development of the SMCT and, by extension, a proficiency-assessment test to measure 
against the standard of the SMCT. 

While the Gansler Commission is the most widely referenced literature pertaining 
to failures in the Army’s acquisition workforce, similar calls for better training of 
personnel had been highlighted by other governmental oversight agencies as well. In 
July 2007, David M. Walker, then Comptroller General of the United States, gave 
testimony before the Committee on Homeland Security and Governmental Affairs of the 
U.S. Senate, highlighting systemic challenges to federal acquisition and contracting. His 
statement, published in GAO Report 07-1098T, Federal Acquisition and Contracting: 
Systemic Challenges Need Attention , highlights four key acquisition challenges affecting 
agencies in the U.S. Government. These challenges are as follows: (1) separating wants 
from needs, (2) establishing and supporting realistic program requirements, (3) using 
contractors in appropriate circumstances and contracts as a management tool, and (4) 
creating a capable workforce and holding it accountable (GAO, 2007). A proficiency- 
assessment test will assist the ECC in creating a capable workforce by testing their 
proficiency against an established standard. 

While the Army has gone to great lengths to correct the deficiencies highlighted 

in the Gansler report (DoA, 2009a), many problems in the training of contracting 

personnel exist. For example, in March 2008, the GAO investigated the Army 

Contracting Agency’s (ACA) Contracting Center of Excellence (CCE) to better 

understand the use of contractors as contract specialists. The primary concern of the 
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GAO was that the Army’s practice of hiring-out contract specialists placed risk on the 
Army of entering into personal service contracts. The GAO identified that the CCE did 
not have a formal training program for its permanent government contracting specialists. 
CCE officials acknowledged the lack of a training program and the need to provide better 
training to their contracting specialists (GAO, 2008, March). 

A similar study highlighting the lack of training in contracting personnel was 
published in June 2009 by the Commission on Wartime Contracting (CWC). The CWC 
was recently established by Congress to investigate fraud, waste, and abuse allegations 
surrounding contingency contracting in a wartime environment. The CWC provided an 
interim report, titled At What Cost? Contingency Contracting in Iraq and Afghanistan , 
that acknowledged the long-standing training problems of the federal civilian and 
military contracting workforce. The report states: 

The ongoing analysis includes a review of the reports related to 
contingency contracting published between FY 2003 and FY 2009. A 
number of themes surface in these reports, such as staffing issues, 
inadequate internal controls, and the need for more training of personnel 
with contracting responsibilities. (CWC, 2009, p. 26) 

The repeated call for more and better training for CCOs by government oversight 
agencies appears to be more prevalent due to the wars in Iraq and Afghanistan. However, 
a literature review of previous training plans and CCO training tasks reveals that the 
development of CCO training has evolved with our military’s missions over the past 
decades. The next portion of the literature review focuses on recommended CCO 
training tasks, which culminated with the development of the U.S. Army’s Soldier’s 
Manucd of Common Tasks. 

4. Formalization of Contingency Contracting Officer Training and 
Tasks 

Until recently, the U.S. military lacked a formalized training program that 
adequately prepared CCOs for contingency contracting operations. In recent years, 
studies have identified the need for formalized training and have presented 
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recommendations on the tasks in which CCOs need training to perform that will best 
prepare them for deployment to a contingency environment. 

Mason (1988) identified the need for formal CCO training in his research. Using 
qualitative data derived from interviewing experienced contracting personnel, Mason 
investigated what, where, and how contingency contracting training was conducted in the 
past. Mason investigated existing legislation, regulations, and policies that affected the 
ability of CCOs to perform their duties, and he looked at potential changes that could be 
made to existing training methods to aid CCOs in carrying out contingency support 
missions. Even though Mason determined that there was a legitimate need for CCOs to 
be trained by their units prior to deployment, he did not present a formal training 
program. Instead, he presented a list of recommendations that focused on institutional, 
procedural and policy changes to better support contingency contracting. 

Killen and Wilson (1992) recommended the following topics for CCOs deploying 
with their units: 

1. Overseas acquisition procedures 

2. Local purchasing 

3. Alternative methods of contracting 

4. The economic variation provision for armed conflict 

5. Base closure at the end of conflict 

This list was the first of its kind in that it recommends special areas of emphasis in order 
for CCOs to be successful in a contingency environment. The list, as limited as it may 
appear today, provided a foundation for building realistic training programs for future 
CCOs (Kirstein, 2003). 

Tigges and Snyder (1993) conducted further research in the field of contingency 
contracting. They presented a list of recommended CCO training topics based on the 
results of their study. Their study focused on the impressions CCOs had of the training 
they received prior to deployment. They conducted a survey and several interviews to 
collect data. Their population of interest was CCOs who deployed during Operation 
Desert Shield. Tigges and Snyder compiled a list of 50 topics and ranked them in 
descending order based on their significance. The following is a list of CCO training 
tasks by rank, as identified by Tigges and Snyder. 
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1. Services 

2. Communications lines 

3. Use of SF 44, Purchase Order and Invoice Voucher 

4. Currency issues 

5. Use of Blanket Purchase Agreements 

6. Host Nation Support Agreements 

7. Commodities 

8. Obtaining a vehicle 

9. Finance issues and procedures 

10. Procurement integrity in a contingency 

11. Cash payments 

12. Dealing with kickbacks 

13. Makeup of local community 

14. Transportation issues and procedures 

15. Customs issues 

16. How to establish a pre-deployment listing of critical requirements 

17. Security issues 

18. Protection of funds under field conditions 

19. Supply issues and procedures 

20. Claims 

21. Mutual support agreements with other nations 

22. Civil engineering issues and procedures 

23. Understanding legal authority under emergency conditions 

24. Cultural issues 

25. Obtaining interpreters 

26. Anticipating customer requests 

27. Chain of command in a contingency 

28. Converting funds 

29. Methods of control 

30. Terminating for convenience 

31. Finding a place to work 

32. Services issues and procedures 

33. Existing vendor source listings 

34. Working with the other branches of the U.S. armed forces 

35. Payments 

36. Dress issues 

37. Referrals to other in-theater organizations 

38. Obligation authority (AF Form 616) 

39. Writing Statements of Work 

40. Home base support 

41. Working with the U.S. Embassy 

42. Restrictive commanders 

43. Establishing review and approval procedures 

44. Record disposition 

45. Legal limitations of decentralization 
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46. Proximity to critical organizations 

47. Appropriation sources 

48. Contract surveillance 

49. Termination for default 

50. Intermediate base support 

Some of these topics are no longer an issue to CCOs, but many have been incorporated 
into training plans and contingency-contracting courses offered to CCOs across all 
branches of the military. 

Nash (1997) identified five skills essential to a contracting officer. These five 
skills, listed below, also apply in a contingency environment. 

1. ) Knowledge of the rules of the game 

2. ) Ability to exercise sound business judgment 

3. ) Knowledge of strategy and tactics 

4. ) Knowledge of the market 

5. ) Ability to function successfully 

For each skill, Nash analyzed the training provided at that time and suggested changes. 
Of the five skills, Nash argued that only one, knowledge of the rules of the game, 
received adequate training. His study suggested that contracting training at that time did 
not fully prepare contracting officers or CCOs for their positions. Furthermore, if 
contracting officers had all five skills, then they probably learned them on the job and not 
through a training program (Nash, 1997). 

Lasch (2002) used a different approach to conduct his study. He identified 88 
tasks that were regularly performed by CCOs in deployed environments. He then 
conducted a survey to obtain feedback from experienced CCOs and Functional Area 
Managers (FAMs) at the major command level (MAJCOM) from the Air Force. The two 
groups were asked to rank the 88 tasks based on how important they believed the tasks 
were and the frequency with which the tasks were performed. The result was a 
composite rank-ordered list of tasks based on inputs from the two surveyed populations 
that considered both the importance and frequencies of the tasks (Lasch, 2002). Lasch 
recommended the top third of the 88 tasks be included in CCO training programs. The 
following is a list of the top 35 training tasks of the 88 identified in the study (Lasch, 
2002 ). 
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1. Simplified Acquisition Procedures 

2. AF Form 9, Purchase Request 

3. Installation Access for Contractors 

4. Customer Education on Contracting Policies 

5. Use of Automated Database/Spreadsheet to Record Purchases 

6. Contract Modifications 

7. SF 44, Purchase Order Invoice and Voucher 

8. Blanket Purchase Agreements 

9. Country Customs Procedures 

10. Contract Modifications 

11. Expedited Contracting Actions 

12. SF 1449, Solicitation/Contract Order for Commercial Items 

13. Shipment of Supplies Overseas 

14. Use of Government Purchase Card 

15. Solicit, Award, and Administer Service Contracts 

16. After-Action Report 

17. Standing Up a Contracting Office 

18. SF 30, Amendment of Solicitation 

19. Deployment/Contingency Kit Contents 

20. Reviewing Statement of Work/Performance Work Statement 

21. Solicit, Award, and Administer Commodity Contracts 

22. Solicit, Award, and Administer Construction Contracts 

23. Host Nation Support Agreements 

24. Contract Closeout 

25. Payments 

26. Commander’s In-brief 

27. Funding the Government Purchase Card 

28. Establishing Vendor Base 

29. AF Form 616, Fund Cite Authorization 

30. Status of Forces Agreement 

31. Working with Finance/Defense Finance and Accounting Services 

32. Writing Statement of Work/Performance Work Statement 

33. Ethics Training 

34. Terminations 

35. Gratuity Training 

There is no evidence that Fasch’s list was incorporated into any CCO training program. 
The list addresses the concerns and challenges CCOs had at that time and provides 
relevant training topics, some of which are valid for the current contingency environment. 

Kirstein’s (2003) work is follow-on research to Fasch’s thesis. Kirstein’s study 
investigates how units conduct training and to what extent the recommended tasks are 
being addressed in training. Kirstein conducted interviews and surveys and reviewed 
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archival training logs to capture the data for his study. His study consisted of a pre-test, 
which was conducted through interviews, and a two-phase survey. Phase one described 
the purpose of the study, and phase two involved the collection of surveys from the 
population of interest. 

Kirstein obtained training logs and plans from the participants’ units. These logs 
provided insight into the content of training and allowed the researcher to investigate 
which of the recommended tasks from Lasch (2002) were included in unit training 
programs across the Air Force (Kirstein, 2003). Kirstein’s analysis found that a number 
of units in some of the commands did not conduct any formal CCO training, clearly 
indicating that many CCOs were not receiving training in the top 30 tasks identified by 
Lasch (2002). Additionally, the units that conducted training did so on approximately 
seven of Lasch’s 30 tasks. 

Kirstein’s review of training logs also revealed that unit-level training alone was 
not preparing CCOs for their deployments. These results indicated that the content and 
quality of training was inconsistent across contingency contracting units. The study 
revealed that most units had nearly complete control over their own training programs. 
While this allowed a great deal of flexibility, there was no way of ensuring the adequacy 
and consistency of training (Kirstein, 2003). Responses to Kirstein’s survey indicated 
that the typical unit conducted unit-level CCO training for one hour each month. The 
training consisted of lectures and was not realistic. Kirstein recommended that the Air 
Force both improve training programs and provide relevant training material for future 
unit-level CCO training programs. The following list of tasks is a list of Kirstein’s 
recommendations. Tasks common to both Lasch’s and Kirstein’s lists are depicted in 
bold below. 

1. SF 44 

2. Blanket Purchase Agreements 

3. Use of Government Purchase Card 

4. Cultural Training 

5. Contract Types 

6. Payments/Funding 

7. Customer Education on Contract Policies 

8. Forms 

9. CCO Responsibilities 
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10. Solicit, Award, and Administer Construction Contracts 

11. AF Form 9/Purchase Orders/Requests 

12. Prioritization 

13. General Procedures 

14. Appendix CC Overview 

15. Working with Finance 

16. Documentation 

17. Inter-service Procedures 

18. Deployment/Contingency Kit Contents/Usage 

19. Solicit, Award, and Administer Service Contracts 

20. Commander’s In-brief 


The above list depicts 20 of the 34 tasks recommended in Kirstein’s study. His 
recommendations were based on feedback from CCOs who had made numerous 
deployments to contingency environments; they continue to be relevant today. When 
combined with the list of tasks recommended by Lasch (2002), the list of tasks 
recommended by Kirstein provides the basics for a good contingency-contracting training 
plan. The methodologies were somewhat different, but the two studies arrived at a 
similar conclusion, with almost half of the recommended tasks being present in both 
studies. 


5. Other Established Training Methods 

The literature reviewed thus far pertains to studies conducted by members of the 
Air Force and the Army as well as the Gansler Commission report, which addressed the 
problems plaguing contingency contracting in the Army. Burger, Kehr and Wobensmith 
(2003) conducted a study sponsored by the Marine Corps Institute (MCI) on how to assist 
the Marine Corps in training its contracting personnel in preparation for deployment to a 
contingency operation. Burger, Kehr and Wobensmith, (2003) identified five causes 
affecting contingency-contracting training in the Marine Corps. These causes are listed 
below. 


Cause 1. No Formal School Training 

Cause 2. No Web-based or Self-guided Study 

Cause 3. No Direct CON 234 Supervision 

Cause 4. No In-house Contingency Contracting Training 

Cause 5. Small Contracting Specialist Population 
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Burger, Kehr and Wobensmith (2003) included elaborate recommendations for 
each one of these causes. For Cause 1, they recommended Marine Corps CCOs attend 
the entry-level Air Force Contracting School. At the time of the study, the Air Force was 
the only Service that had a formal technical school where enlisted personnel could be 
trained in the contracting career field. Attendance to the Air Force Contracting School 
would allow enlisted CCOs to get training in contract policies, best practices, and 
guidelines to perform their jobs. For Cause 2, the study group recommended the 
development of a contingency-contracting correspondence manual that they, in 
conjunction with the MCI, developed and published as part of their report. For Cause 3, 
the group recommended a more proactive role to ensure that all CCOs attended CON 
234, Contingency Contracting Course. They recommended that instead of only tracking 
who completed the course, the Logistics Branch at Headquarters Marine Corps 
(HQMC)—the branch of the Marine Corps that oversees the USMC CCOs—should take 
a more centralized approach to managing the available slots and ensuring more CCOs 
attend CON 234. For Cause 4, the group recommended the establishment of a Mobile 
Training Team (MTT) made up of experienced CCOs. The team would travel to the 
locations where the majority of CCOs were located and provide contingency contracting 
training. 


6. Defense Acquisition University CON 234, Contingency Contracting 
Course 

The Defense Acquisition University (DAU) provides acquisition training and 
career management to the contingency contracting and procurement community. The 
DAU mission is to “[pjrovide practitioner training, career management, and services to 
enable the Acquisition Technology and Logistics (AT&L) community to make smart 
business decisions and deliver timely and affordable capabilities to the war-fighter” 
(DAU, 2008). The DAU offers 102 formal courses, 22 of which are in the field of 
contracting; however, only CON 234 addresses contingency contracting. 

CON 234 is a 9-day course intended to train and develop skills in contingency 
contracting to support joint forces in all military operations (DAU, 2008). While some 
CCOs who have attended CON 234 categorize it as a great course that goes a long way in 
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preparing CCOs for deployment to a contingency environment, others complain that the 
course is too short and too broad (Burger et al., 2003). 


7. Contingency Contracting Officer Handbook 

Contingency Contracting: A Joint Handbook for the 21st Century was prepared 
by the Office of the Under Secretary of Defense for Acquisition, Technology and 
Logistics (OUSD (AT&L)), Defense Acquisition, Technology, and Logistics 
contingency-contracting staff; the Defense Acquisition University; representatives from 
the United States Army, Navy, Marine Corps, and Air Force; and the Air Force Logistics 
Management Agency (DPAP, 2008). The CCO handbook serves as a guide to assist 
contingency-contracting officers in preparing for wartime and peacetime deployments or 
exercises and performing the duties of a deployed contingency contracting officer. 

8. Soldier’s Manual of Common Tasks 

The Soldier’s Manual of Common Tasks (SMCT) was created by the 
Expeditionary Contracting Command (ECC) for soldiers and officers holding Military 
Occupational Specialty (MOS) 51C as a way to focus on specific tasks that will prepare 
CCOs in the ECC for their wartime mission. The SMCT contains an MOS training plan 
that provides information needed to plan, conduct, and evaluate individual training (DoA, 
2009, September 22). The SMCT includes a list of 36 tasks divided into four subject 
areas that are designed to prepare CCOs to accomplish their mission in any contingency 
environment. Chapter III of this report describes the development of the SMCT and the 
establishment of the ECC in greater detail. 

The establishment and standardization of training tasks for CCOs in the SMCT 

will help subordinate units and individual CCOs within the ECC focus their training 

efforts prior to deployment. This body of knowledge is a significant step in standardizing 

the pre-deployment training for CCOs and preparing them for their mission once 

deployed. To supplement this training, an appropriate performance assessment tool is 

needed to measure performance of these tasks. The ECC currently lacks a standardized 

method of measuring individual skill and task proficiency in its CCOs. A written 

proficiency-assessment test will allow leaders in the ECC to measure individual task 
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proficiency against established standards, determine each CCO’s preparedness to conduct 
duties during deployment, determine appropriate CCO assignment, and develop 
individualized training programs for CCOs. 

Developing a proficiency-assessment test requires extensive knowledge of test- 
construction theory and practices. The research team, comprised of professional soldiers, 
has no academic background or experience in test construction. However, training 
soldiers and assessing their performance is a core competency of any professional Army 
officer. In this respect, the research team has practical experience. In an effort to round 
out our practical experience with grounded theoretical and academic principles and to 
assist in the development of the PAT, the research team reviewed U.S. Army training 
doctrine, as it relates to training and assessment. Additionally, the research team 
conducted a review of assessment as a subset of the adult-learning model, core test 
principles, test construction and test blueprints, Bloom’s Taxonomy, and item writing, as 
it relates to test construction. This knowledge base will assist the research team in 
developing a valid and reliable proficiency-assessment test to meet the needs of the 
Expeditionary Contracting Command and U.S. Army CCOs. 

C. TESTING AND EVALUATION 

This portion of the literature review will primarily review test and assessment 
theory. At the beginning of our research, the research team set out to determine (1) what 
is the role of assessment in a professional workforce, (2) what are core test principles, 
and (3) what are the basic steps and procedures to test construction. 

1. Training and Assessment 

Field Manual (FM) 7-0 (Training the Force), FM 7-l(Battle Focused Training), 
and Army Regulation 350-1 (Army Training and Feader Development) are three of the 
main documents that outline Army training doctrine. FM 7-0 establishes Army training 
doctrine while FM 7-1 is designed to help unit commander’s develop and execute 
training programs (DoA, 2002). The authors reviewed those documents to understand 
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how this project fits within the Army training methodology and comparative adult¬ 
learning theory. The Army training and leader-development model is depicted below in 
Figure 1. 



Figure 1. Army Training and Leader Development Model 

(DoA, 2002, p. 1-6) 


FM 7-0 describes this model in the following manner: 


Leader Development is a lifelong learning process. The three 
core domains that shape the critical learning experiences 
throughout a soldier’s and leader’s career are the operational, 
institutional, and self-development domains. Together, these 
domains interact using feedback and assessment from various 
sources and methods to maximize warfighting readiness. Each 
domain has specific, measurable actions that must occur to 
develop our leaders. The operational domain includes home 
station training, combat training center rotations, joint training 
exercises, and operational deployments that satisfy national 
objectives. Each of these actions provides foundational 
experiences for soldiers, leaders, and unit development. The 
institutional domain focuses on educating and training soldiers 
and leaders on the key knowledge, skills, and attributes required 
to operate in any environment. It includes individual, unit and 
joint schools, and advanced education. The self-development 
domain, both structured and informal, focuses on taking those 
actions necessary to reduce or eliminate the gap between 
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operational and institutional experiences. Throughout this 
lifelong learning and experience process, there is formal and 
informal assessment and feedback of performance to prepare 
leaders for their next level of responsibility. Assessment is the 
method used to determine the proficiency and potential of 
leaders against a known standard. (DoA, 2002, p. 1-5) 

The proficiency-assessment tool developed in this project can help assess and coordinate 
individual training within the institutional and self-development domains of the Army 
training model. 

Assessment is core to adult-learning processes. The adult-learning model 
includes four phases (Knowles, Holton, & Swanson, 1998). The first phase is to 
determine what learning objectives are desired and to set learning achievement goals. 
The second phase creates a strategy and allocates resources to accomplish the learning 
objectives. Step three involves implementing the learning strategy, and step four assesses 
the attainment of the learning goal. Figure 2 depicts the theoretical foundation of adult 
learning. 



Figure 2. Theoretical Foundation of Adult Learning 
(Knowles, Holton & Swanson, 1998, p. 125) 
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Army training doctrine considers assessment a commander’s responsibility (DoA, 
2002). It is ultimately the commander who must determine if her unit and soldiers are 
prepared to accomplish their operational mission. The commander utilizes multiple 
evaluation tools to determine and report mission readiness. These evaluations can be 
classified as informal, formal, internal, or external. 

Assessment is a continuous process and is considered both the end and the 
beginning of the training-management cycle. The Army’s planning process for training 
is divided into four steps: training assessment, training strategy, commander’s guidance, 
and training plans. Figure 3 depicts the Army’s planning process for training. 



Figure 3. Army Training Planning Process (Training Assessment) 

(DoA, 2003, p. 4-4) 

Before developing a training plan, a commander evaluates the individual battle 
tasks that support overall unit mission essential tasks. Mission Essential Task Lists 
(METL) are collective tasks that combine individual and leader battle tasks. Tasks within 
the 51C SMCT are examples of individual tasks that support these overall mission- 
essential tasks. Individual- and mission- essential tasks are rated in FM 7-1 as follows: 
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ASSESSMENT RATINGS 


The commander’s training assessment is required for each METL 
task. Task proficiency is rated as— 

“T” (trained): The unit is trained and has demonstrated proficiency in 
accomplishing the task to the Army standard. The leader judges task 
performance to be free of significant shortcomings. Training on “T” 
tasks is designed to sustain proficiency on that task. 

“P” (needs practice): The unit can perform the task with some 
shortcomings. Performance has demonstrated that the unit does not 
achieve the standard without some difficulty or has failed to perform 
some task steps to standard. The shortcomings are not severe enough 
to require complete retraining. Only refresher training is required. 

“U” (untrained): The unit cannot demonstrate an ability to achieve 
wartime proficiency. The leader prepares a comprehensive plan to 
train all supporting tasks not executed to standard. Unless the task is a 
new METL task, a rating of “U” indicates a serious training 
deficiency and reflects on the unit’s wartime readiness posture. 

EVALUATION RATINGS 

Evaluation ratings are given for specific task proficiency and should 
not be confused with leader assessments. Evaluation ratings are 
ratings assigned directly to the performance of a specific task or 
component steps of a task. The standard evaluation ratings, discussed 
further in chapter 6, are as follows: 

“GO”: The task or performance step of a task was performed to 
standard. A rating of GO is normally awarded if all steps in the task 
are passed. 

“NO GO”: The task or any performance step in the task was not 
performed to standard. (DoA, 2003, p. 4-6) 

How commanders develop a training plan and evaluate individual tasks is relevant 
to this project because this reveals ways commanders might consider using the 
proficiency-assessment test developed in this project. It is not the research team’s intent 
to develop an assessment tool for commanders to automatically determine specific battle- 
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task ratings as defined above. By understanding this construct, the research team was 
able to write specific and relevant test instructions. 

It is the hope of the research team that the assessment tool developed in this 

project will satisfy some of the requirements outlined in Army training doctrine to 

establish individual self-development programs. The following excerpt from FM 7-0 

describes the role of assessment in establishing individual self-development programs: 

Self-development starts with an assessment of individual strengths, 
weaknesses, potential, and developmental needs. Commanders and 
leaders provide feedback that enables subordinates to determine the 
reasons for their strengths and weaknesses. Together, they prioritize 
self-development near-term and long-term goals and determine 
courses of action to improve performance. Self-development is— 

A planned process involving the leader and the subordinate being 
developed. It enhances previously acquired skills, knowledge, 
behaviors, and experience; contributes to personal development; and 
highlights the potential for progressively more complex and higher- 
level assignments. Self-development focuses on maximizing 
individual strengths, minimizing weaknesses, and achieving 
individual personal and professional development goals. 

Initially very structured and generally narrow in focus. The focus 
broadens as individuals understand their strengths and weaknesses, 
determine their individual needs, and become more experienced. Each 
soldier’s knowledge and perspective increases with experience, 
institutional training, and operational assignments, and is accelerated 
and broadened by specific, goal-oriented self-development actions. 

(DoA, 2003, p. 1-12) 


The PAT developed during this research can be used by individual soldiers—and 
their leaders and commanders—to conduct an assessment of that soldier’s strengths and 
weaknesses in the SMCT tasks. These assessments can then be used to design follow-on 
training. The PAT can measure the success of this follow-on training designed as a result 
of that assessment. 

The six core adult-learning principles should be considered when establishing a 
self-development program (Knowles, Holton & Swanson). Figure 4 depicts the six core 
adult learning principles: 
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Figure 4. Core Adult Learning 
(Knowles, Holton & Swanson, 1998, p. 4) 


Knowles, Holton and Swanson expand upon the concept of the adult-learning 
model and provide eight steps to guide field supervisors and subordinates to develop a 


learning contract. These steps are as follows: 


1. Diagnose your learning needs, 

2. Specify your learning objectives, 

3. Specify learning resources and strategies, 

4. Specify evidence of accomplishment, 

5. Specify how evidence will be validated, 

6. Review the contract, 

7. Carry out the contract, and 

8. Evaluation of your learning. 
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A learning contract should be oriented in such a way that the learner assumes a 
major responsibility to ensure that the contract is relevant (Piskurich, Beckschi & Hall, 
2000). Piskurich et al. provides the following reasons to justify the increased use of 
learning contracts to satisfy training and development objectives: 

1. To provide more appropriate learning for individuals with diverse 
backgrounds, experiences, and learning styles; 

2. To meet the needs of learners in specialized areas; 

3. To meet needs of learners in rapidly changing fields when no 
appropriate curriculum or training is available; 

4. To meet the needs of learners at a distance; 

5. To save training dollars; and 

6. To develop self-directed, reflective, continuing learners who can 
contribute to the success and growth of the organization. 


Several methods and models are available for supervisors to evaluate implemented 
training programs (Phillups, 1997; Piskurich, Beckschi & Hall, 2000). Assessment 
should occur at every level of training development. 


This section of the literature review examined how assessment fits within training 
and leadership development models, both inside and out of the military. Additionally, we 
reviewed relevant information in developing learning contracts and adult learning 
principles. This information is relevant for establishing a context of how assessments are 
used towards self-development programs as well as for serving as a possible guide for the 
research team during the development and publication of the commander’s assessment- 
tool guide. 


The literature review until now has served in providing the research team with the 
historical background of contingency-contracting training and where a test would fit in a 
training methodology. The following sections of the literature discuss the core test 
principles, test construction methods, test blueprints and item writing. These sections 
deal directly with the mechanics of writing the test. This section provided the research 
team with the knowledge of how to develop and write the questions for the PAT. 
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2. Core Test Principles 

Tests can be categorized into numerous types. Examples of these categories are 
intelligence, ability, personality, objective, projective, and attitude, among others (Kline, 
1986). Tests are specific tools designed and administered to serve a specific function. 
These functions can be categorized into three areas: prediction, assessment, and trait 
measurement (Nunnally, 1972). Tests designed to a predictive function attempt to 
forecast how an individual will behave in a particular situation. An example of this test 
might be a college placement exam. Assessment tests do not attempt to predict future 
behavior, but rather directly measure the current performance of an individual at a 
specific point in time. The level of performance measured depends upon the type of test 
developed and administered. Trait-measurement tests are designed to measure specific 
psychological traits such as anxiety or depression. 

Another way to categorize tests is to classify them as either a criterion-referenced 
test or a norm-referenced test (Marshall & Hales, 1972). Criterion-referenced tests 
attempt to determine if an individual has mastered a subject area in a specific learning 
objective. This measurement could be interpreted in a variety of ways, such as deciding 
if an individual is ready to move to another subject or higher level of mastery. This type 
of test is often used as a diagnostic instrument. Norm-referenced tests attempt to measure 
an individual against a referenced norm, such as a group of colleagues or other 
professionals. This type of testing attempts to rank students on a scale of magnitude 
within the distribution of a specific population. The proficiency-assessment test 
developed within the scope of this project can be classified as a criterion-referenced test. 

Tests involve three key players: the test user, test writer, and test taker. The 
Standards for Educational and Psychological Tests defines a test user as one who 
chooses tests to give, interprets scores, or makes decisions based on test scores. 
Dependent upon the use or function of the test, the following standards are considered 
essential for test users (APA, 1974, p. 58-65): 

1. Test user should have a general knowledge of measurement 
principles and of the limitations of test interpretations, and 
how well they match the qualifications required to use a 
specific test. 
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2. A test user should know and understand the literature 
relevant to the test she uses. 

3. One who has the responsibility for decisions about 
individuals or policies that are based on test results should 
have an understanding of psychological or educational 
measurement. 

4. Principal test users within an organization should make 
every effort to be sure that all those in the organization who 
are charged with responsibilities related to test use and 
interpretation have received training appropriate to those 
responsibilities. 

5. Tests users should seek to avoid bias in test selection and 
interpretation or even the appearance of discriminatory 
practice. 

6. Institutional test users should establish procedures for 
periodic internal review of test use. 

7. The choice or development of tests or other assessment 
procedures should be based on clearly formulated goals and 
hypothesis. 

8. A test user should consider more than one variable for 
assessment and the assessment of any given variable by 
more than one method. 

9. In choosing an existing test, a test user should relate its 
history of research and development to his intended use of 
the instrument. 

10. A test user is expected to follow carefully the procedures 
described in the test manual. 

11. The test administrator is responsible for establishing 
conditions, consistent with the principle of standardization, 
that enable each examinee to do his or her best. 

12. The test user shares with the test developer or distributor a 
responsibility for maintaining test security. 

13. A test score should be interpreted as an estimate of 
performance under a given set of circumstances. It should 
not be interpreted as some absolute characteristic of the 
examinee or as something permanent. 


Anastasi (1982) recommends that two main reasons to control the use of a test is to 
ensure that it is administered by a qualified examiner and to protect the test content from 
being compromised, which would invalidate the assessment. A qualified examiner must 
essentially know how to select the right test, administer the test, and properly interpret the 
results. 
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Every text on measurement theory and test construction that the research team 
reviewed, listed reliability and validity as two essential test characteristics. Some 
references added additional characteristics such as objectivity, balance, fairness, and 
practicality to the list (Marshall & Hales, 1972). Another author considers efficiency as a 
characteristic and highlights current theories and models designed to improve test 
efficiency (Brennan, 2006). For simplicity, this literature review focuses on reliability 
and validity as the two main test characteristics. 

Reliability, in essence, measures the consistency of a test. The more consistent a 
test is, the higher the probability the same individual will obtain a similar score when 
retesting on an identical test or equivalent version (Anastasi, 1982). Statistically, there 
are multiple methods to derive reliability: 

Reliability can be defined in several ways—such as the proportion of 
observed-score variance that is true-score variance, the squared 
correlation between true scores and observed scores, or the correlation 
between observed scores on two parallel tests. Several estimates of 
reliability are widely used, including test/retest, parallel-forms, 
alternate forms, and internal-consistency estimates. (Allen & Yen, 

2001, p. 91) 

Test developers want to minimize the effects of chance on reliability (Wood, 1960). 
Wood identifies the following three sources of chance variation: 

1. Scorer unreliability occurs when the scorer through subjectivity or bias induces 
variability into an assessment or testing process. 

2. Content unreliability occurs from a poor sampling of content. It typically occurs 
where a score of an individual depends heavily on the sample of questions 
selected from a test speculation. 

3. Temporal unreliability includes a variety of factors that would explain how an 
individual’s score varied at different times. One example of this is the varying 
ability of an individual to remember responses from the last test when taking the 
second test. 


According to Kline (1996), other recommended sources of unreliability include guessing, 
test length, and poor test instructions, or simply a sick or ill test subject. 
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Validity is how well a test meets its designed function. A test might be extremely 
reliable, yet at the same time, be invalid. It is never acceptable to accept a test as valid 
until something proves it as such (Nunnally, 1972): 

Validity can be assessed in several ways, depending on the test and its 
intended use. The three major types of validity are content validity, 
criterion-related validity and construct validity. Determinations of 
criterion-related validity and construct validity involve the calculation 
and examination of correlations or other statistics. Content validity, 
however, does not involve any statistical calculations. (Allen & Yen, 

2001, p. 95) 

Content validity is achieved through an individual, subjective judgment. 
Essentially, it is established when someone examines the test and determines it correctly 
measures what it was designed to measure. This type of validity is more subject to error 
and should not be relied upon as the sole determination of a test’s validity (Allen & Yen, 
2001). The research team recognizes the limitations of content validity, but we consider 
it the most achievable type of validity within the scope of the project and a primary 
concern during test construction. Nunnally & Bernstein (1978) discuss content validity 
in the following way: 

Rather than test the validity of measures after they are constructed, 
one should ensure validity by the plan and procedures of construction 
[...]. The validity of the measure is judged by the character of the 
plan and by the apparent skill with which the plan has been carried 
out. If it is agreed by most potential users of the test, or at least by 
persons in positions of responsibility, that the plan was sound and 
well carried out, the test has a high degree of content validity. 

The simple example above illustrates the two major standards for 
ensuring content validity: (1) a representative collection of items and 
(2) “sensible” methods of test construction, (pp. 80-81) 

Criterion-related validity is derived by establishing a correlation between a test 
score and a specific criterion’s score. Criterion-related validity is interchangeable with 
the term predictive validity (Nunnally & Bernstein, 1978). Brenan (2006) describes the 
evolution of all three models, the limitations of each model, and the methods of 
estimation. It is not the research team’s intention to describe the science of validity in 
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detail, but rather to identify it as a core test principle and recognize its relevance towards 
the assessment tool developed in this project. 

The Standards for Educational and Psychological Tests provides essential 
standards for test developers to achieve both reliability and validity. Many of these 
standards are achieved throughout the entire test-construction process. Steps are taken by 
test creators to reevaluate validity and reliability using test trials and statistical analysis 
before a test is published in its final form. 

3. Test Construction 

This section of the literature review examines recognized steps of test 
construction and identifies rules and procedures for some of the key steps. Allen & Yen 
(2001) identify the following basic steps in test development: 

1. Plan the test. During this step, developers establish a test map or test 
blueprint to document what areas the test items will cover. 

2. Write the test items or questions. 

3. Administer all written test items to a sample population. Generally, a 
minimum of 50 persons is reasonable, but several hundred is optimal. 
The sample population should represent the population for which the 
test was developed. 

4. Conduct item analysis using the results of the sample. During this 
step, test developers refine test items and eliminate those that are 
invalid. 

5. Administer the revised tests to another sample population and continue 
steps 2-4 until a satisfactory test is produced. 

Downing and Haladyna (2006), list twelve steps for effective test development: 

1. Overall plan, 

2. Content definition, 

3. Test specifications, 

4. Item development, 

5. Test design and assembly, 

6. Test production, 

7. Test administration, 

8. Scoring test responses, 

9. Passing scores, 

10. Reporting test results, 

11. Item banking, and 

12. Test technical report, (p. 5) 


42 



Building a test blueprint and item writing are two of the common steps found between 
these test development models that we intend to execute within the scope of this project. 
What follows in this literature review is a more detailed look at these steps, and in 
Chapter IV of this project, we identify what procedures we followed to develop the 
assessment tool. 

4. Test Blueprints 

Test blueprints act as a road map for test construction in laying out which subject 
areas the test developer wants to test and at what cognitive domain the subject area 
should be tested. 

Test specifications are often called blueprints because they specify 
how the test or pool is to be constructed. [...] Specifications for a 
test form or pool describe its content, form, and functional 
requirements. (Brennan, 2006, p. 315) 

At a minimum, test blueprints or specifications should contain the following information 
(Downing & Haladyna, 2006): 

1. The type of testing format to be used (selected response versus constructed 
response), 

2. The total number of test items as well as the type or format of test items, 

3. The cognitive classification system to be used, 

4. Whether or not the test items or performance prompts will contain visual 
stimuli, 

5. The expected item scoring rules, 

6. How test scores will be interpreted (norm or criterion-referenced), and 

7. The time limit for each item. (p. 9) 


Subject-matter experts and administrative officials should review test blueprints prior to 
implementation (Nunnally & Bernstein, 1978). 

Several texts recognized the use of Bloom’s Taxonomy of Educational Objectives 
as a common method to classify cognitive domains within a test blueprint (Downing& 
Haladyna, 2006; Brennan, 2006; Marshall & Hales, 1972; Kline, 1986). (Brennan, 2006) 
(Kline, 1986) The research team uses this taxonomy during the test-specification step, so 
a brief summary of Bloom’s Taxonomy is included here. 
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Bloom developed a taxonomy of educational objectives in order to assist teachers, 
professionals, and researchers in categorizing educational goals and to guide them with 
curricular and evaluation problems (Bloom, 1956). His work divided the cognitive 
domain into six categories: knowledge, comprehension, application, analysis, synthesis, 
and evaluation. He provided examples of illustrative test items within each domain. His 
classic work was revised by several of his pupils (Anderson et al., 2002). Their revised 
version of Bloom’s taxonomy is depicted in Figure 5 and 6. Anderson et al. establishes a 
knowledge dimension that is divided into four categories—factual, conceptual, 
procedural, and meta-cognitive—and suggests using a taxonomy table as an assessment- 
development tool. Other authors demonstrate the use of similar matrices to develop 
blueprints within a taxonomy construct (Marshall & Hales, 1972; Downing & Haladyna, 
2006). 
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CATEGORIES 

& cognitive Alternative 

PROCESSES NAMES 

Definitions and Examples 

1. remember —Retrieve relevant knowledge from long term memory 

1 ’- 1 

RECOGNIZING 

Identifying 

Locating knowledge in long-term memory that is consistent 
with presented material (e.g.. Recognize the dates of 
important events in U.S. history) 

1.2 

RECALLING 

Retrieving 

Retrieving relevant knowledge from long-term memory 
(e.g., Recall the dates of important events in U.S. history) 

J- " 

2. understand —Construct meaning from instructional messages, including oral, written, and 

graphic communication _ 

2.1 

INTERPRETING 

Clarifying, 

paraphrasing, 

representing, 

translating 

Changing from one form of representation (e.g., numerical) 
to another (e.g., verbal) (e.g.. Paraphrase important speeches 
and documents) 

2.2 

EXEMPLIFYING 

Illustrating, 

instantiating 

Finding a specific example or illustration of a concept or prin¬ 
ciple (e.g.. Give examples of various artistic painting styles) 

2.3 

CLASSIFYING 

Categorizing, 

subsuming 

Determining that something belongs to a category (e.g., 
concept or principle) (e.g.. Classify observed or described 
cases of mental disorders) 

i 24 

SUMMARIZING 

Abstracting, 

generalizing 

Abstracting a general theme or major point(s) (e.g., Write a 
short summary of the events portrayed on a videotape) 

2.S 

INFERRING 

Concluding, 

extrapolating, 

interpolating, 

predicting 

Drawing a logical conclusion from presented information 
(e.g., In learning a foreign language, infer grammatical 
principles from examples) 

2.6 

COMPARING 

Contrasting. 

mapping. 

Detecting correspondences between two ideas, objects, and 
the like (e.g.. Compare historical events to contemporary 

I 


matching 

situations) 

2.7 

Explaining 

Constructing 

models 

Constructing a cause-and-effect model of a system (e.g.. Ex¬ 
plain the causes of important 18th-century events in France) 

3. 

apply —Cam out or use a procedure in a given situation 

--------— —- 


| 3.1 executing Carrying out Applying a procedure to a familiar task (e.g.. Divide one 
1 whole number by another whole number, both with 

multiple digits) 


.3.2 implementing Using Applying a procedure to an unfamiliar task (e.g.. Use New¬ 

ton's Second Law in situations in which it is appropriate) 


Figure 5. Revised Bloom’s Taxonomy of Learning Objectives 

(Anderson et al., 2002, p. 66) 
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Categories 

& Cognitive Alternative 

Processes Names 

Definitions and Examples 

4. analyze — Break material into its constituent parts and determine how the parts relate to one 
another and to an overall structure or purpose 

4.1 Differentiating Discriminating, 
distinguishing, 
focusing, 
selecting 

Distinguishing relevant from irrelevant parts or impor- j 
tant from unimportant parts of presented material 
(e g . Distinguish between relevant and irrelevant 
numbers in a mathematical word problem) 

4.2 Organizing 

Finding 

coherence, 

intergrating, 

outlining, 

parsing, 

structuring 

Determining how elements fit or function within a 
structure (e g., Structure evidence in a historical 
description into evidence for and against a particular 
historical explanation) 

4.3 Attributing 

Deconstructing 

Determine a point of view, bias, values, or intent under¬ 
lying presented material (e.g.. Determine the point of 
view of the author of an essay in terms of his or her 
political perspective) 

3. EVALUATE— Make 

judgments baaed on criteria and standards 

5.1 CHECKING 

Coordinating, 

detecting, 

monitoring, 

testing 

Detecting inconsistencies or fallacies within a process or 
product; determining whether a process or product has 
internal consistency, detecting the effectiveness of a pro¬ 
cedure as it is bemg implemented (e g., Determine if a 
scientist's conclusions follow from observed data) 

5.2 CRITIQUING 

Judging 

Detecting inconsistencies between a product and exter¬ 
nal criteria, determining whether a product lias exter¬ 
nal consistency; detecting the appropriateness of a pro¬ 
cedure for a given problem (e.g , judge which of two 
methods is the best way to solve a given problem) 

6. create— Pul elements together to form a coherent or functional whole, reorganize elements 
into a new pattern or structure 

6.1 GENERATING 

Hypothesizing 

Coming up with alternative hypotheses based on 
criteria (e.g.. Generate hypotheses to account for an 
observed phenomenon) 

6.2 Planning 

Designing 

Devising a procedure for accomplishing some task (e.g.. 
Plan a research paper on a given historical topic) 

6.3 Producing 

Constructing 

Inventing a product (e.g.. Build habitats for a specific 
purpose) 


Figure 6. Revised Bloom’s Taxonomy of Learning Objectives 

(Anderson et al., 2002, p. 67) 


5. Item Writing 

Items, or rather questions, can be divided into two general formats (Osterlind, 
1997). Selected-response items occur when the examinee is given both the problem and 
the correct solution. The examinee must then choose the correct solution over other 
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incorrect solutions provided as options. Figure 7 depicts an example of this type of 
format and gives the reader an idea of test-item nomenclature. 


Use the Illustration and road the passage below to answer Direction 
the question. —* 



Graphic 


Charley did an experiment in which he floated an 
eyedropper in a jar of water, He left the jar on a shell tor 
one hour. Then he recorded the level o! water in the 
eyedropper. Next, he heated the jar containing the 
eyedropper tor 30 seconds. 

Which variable M Charley change in this experiment? 


Text 


] 


Stem 


A. kind of eyedropper 

B. level of water tn the |ar 

C. amount of time between temperature measurements 
•D temperature of the air and wafer in the jar 


Detractors 

Correct 

Response 


Figure 7. Example of Selected-response Item and Test-item Nomenclature 

(Osterlind, 1997, p. 34) 


Constructive-response style items only provide the examinee the problem. The 
examinee is required to respond by providing the perceived correct response. Each 
format has its advantages and disadvantages. For example, it is typically easier and less 
resource intensive to score selected-response items. Items can further be refined into 
specific formats such as multiple choice, true/false, matching, sentence completion, short 
answer, essay, and so forth. 

There are two types of test-item writers: professional and casual (Osterlind, 
1997). As casual writers, we used collected checklists to guide us during the item¬ 
writing step. Many of the references we reviewed contained useful item-writing tips. 
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These tips not only focused on the specifics of writing one test item but also included 
more expansive topics, such as how to order items within a test and ethical or legal 
concerns for using copyrighted test items. 

Faulty test items are typically ambiguous and trivial (Nunnally & Bernstein, 
1978). Item writers should establish a random system to position the correct answer 
among alternatives in order to mitigate biases within the test (Wood, 1960). Five rules in 
writing good multiple-choice items include (Kline, 1986) the following: 

1. Simplicity (in writing), 

2. All distracters should be capable of distracting subjects, 

3. Only one option should be correct, 

4. The answer to one question should not give clues to the answer to another, 
and 

5. Avoid testing the trivial because it is easy to test. 

Kline (1986) further suggests that one technique for determining which item 
format best suits a particular subject is to write them all. Through this process, a test 
writer can eliminate those formats that do not suit a particular subject and then select 
from the remaining options the format that best achieves the intended purpose of the 
item. A more exhaustive list of 31 general item-writing guidelines can be found in 
Developing and Validating Multiple-Choice Test Items. Some of these guidelines include 
the following: avoid using “all of the above,” avoid trick items, make the stem as brief as 
possible, place the main idea of the item in the stem and not in the choices, and so forth 
(Haladyna, 2004). Chapter IV, Methodology, clearly describes which rules we used to 
guide us through item writing. 

D. SUMMARY 

This chapter reviews related studies, reports, articles and other literature on the 
topics of contingency contracting and test construction and evaluation. Within the realm 
of contingency contracting, the research team focused on the history and evolution of 
tasks related to training CCOs for deployment to a contingency environment. The review 
examined literature pertaining to contingency contracting on all the major conflicts the 
United States military has taken part in, training methods and tasks for contingency 
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contractors, and the problems plaguing contingency contracting, as highlighted by the 
Gansler report. Our review of contingency contracting culminated with a discussion of 
the establishment of the ECC and the development of the SMCT. Within our discussion 
of test construction and evaluation, the research team focused on overarching principles 
of assessment as a subset of the adult-learning model, core test principles, test 
construction and test blueprints. Bloom’s Taxonomy, and item writing, as it relates to test 
construction. The SMCT, developed to provide CCOs and their leaders with a standard 
by which to train, lacks a standardized test for measuring performance against that 
standard. The research team developed a test by combining these two distinct disciplines 
using knowledge gained during this literature review. 

In conducting this literature review, we realize that there are certain limitations on 
our ability to develop a comprehensive assessment test for CCOs. The most obvious is 
that we are not professional test developers. The field of test construction and evaluation 
is extremely broad and quite extensive. Professional test developers could rightfully take 
years to develop an assessment test for CCOs, while this research has to be completed 
within certain time constraints to satisfy academic requirements. Given our limitations in 
both experience as test developers and in the time constraints of the academic schedule, a 
modified test-development plan was developed that applies the principles of test 
construction and evaluation. Chapter IV, Methodology, covers in detail the actual steps 
we took in developing the PAT and how it varies from the theoretical test-construction 
principles discussed in this chapter. 

The next chapter focuses on frameworks of the contingency-contracting 
environment. The research team focuses primarily on the historical context and 
establishment of the U.S. Army’s Expeditionary Contracting Command (ECC). The 
chapter further discusses the current MOS 51C development and training model and how 
the Soldier’s Manual of Common Tasks relates to this model. 
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III. CONTINGENCY CONTRACTING FRAMEWORK—THE ECC 

AND THE SMCT 

A. INTRODUCTION 

The purpose of this chapter is to provide an overview of the ECC as an 
organization and to describe in detail the content of the SMCT. In this chapter, we 
discuss the context under which the ECC was established and describe the ECC 
organizational structure and training methodology. We also describe how the SMCT fits 
in to the ECC’s training strategy and breakdown the tasks in the document by their 
subcomponent parts. 

B. ESTABLISHMENT OF THE ECC 

The U.S. Army’s Expeditionary Contracting Command (ECC) was established in 
January 2008 in direct response to the Gansler Commission’s recommendations to 
prepare, train and provide skilled contracting personnel in support of U.S. Army 
contingency operations (DoA, 2009a). Among the Gansler Commission’s four 
overarching recommendations, the first two addressed the need for changes in 
institutional and organizational structure to fix many of the Army’s failures identified in 
the report. The first one recommends an increase in personnel, and the second, 
recommends the establishment of the parent organization of those personnel. 
Specifically, recommendation number two, restructure organization and restore 
responsibility to facilitate contracting and contract management in expeditionary and 
CONUS operations, lists several sub-recommendations. Among these are the 
recommendation to establish the Army Contracting Command (ACC) and the 
Expeditionary Contracting Command (ECC) as a subordinate element of the ACC 
(Gansler, 2007). Figure 8 depicts the ACC’s and ECC’s organizational structure. 
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Figure 8. ACC and ECC Organizational Structure 
(ECC, 2009, October 14) 

Recommendation number one, increase the stature, quantity, and career 
development of the Army’s contracting personnel, both military and civilian (especially 
for expeditionary operations), recommends the authorization for general-officer billets to 
lead those organizations, the establishment of MOS 51C, and increasing the Army’s 
contracting workforce by 400 uniformed personnel and 1,000 civilian personnel (Gansler, 
2007). It is these increases in uniformed personnel will eventually fill the ranks of this 
new organization. While the Gansler Commission provided the recommendations for 
these new changes, General Order (GO) No. 2009-20, published July 8, 2009, officially 
established the ACC and ECC, effective January 30, 2008 (DoA, 2009, July 8). 
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C. ECC ORGANIZATIONAL STRUCTURE 

The ECC is comprised of a command headquarters and six contingency support 
brigades (CSBs). These CSBs are aligned with each Army geographical theater. Each 
CSB is comprised of one or more contingency-contracting battalion (CCBn), which, in 
turn, generally contains two or more contingency contracting teams of four MOS 51C 
personnel. The ECC headquarters provides the command and control of the Army’s 
expeditionary-contracting assets. It is responsible for training, equipping, and providing 
trained contracting capability to support any contingency (ECC, 2009, October 14). It is 
under this charter that the ECC has developed the SMCT , with the intent of establishing a 
measurable standard for CCOs to train against in preparation for deployment. The lack 
of an assessment test to measure against that standard necessitates the development of the 
PAT and is the main driver for this research project. 

D. TRAINING METHODOLOGY 

In developing a training methodology for CCOs in this new organization, the 
ECC acknowledges the multitude of requirements of a newly assigned CCO. Figure 9 
depicts the ECC’s CCO Development Model, which shows the multitude of requirements 
CCOs face to become proficient. This model shows the SMCT as one of the many 
educational and professional-development requirements of a CCO. At the core of a 
CCO’s training is his warrior-task training. CCOs must remain proficient in the 
warfighting tasks of shooting, moving, and communicating in order to survive and win on 
the battlefield. Additionally, a CCO must take part in mandatory military training, such 
as training exercises at the training centers. As an acquisition professional, the CCO 
must meet civilian educational requirements as well as DAU certification requirements to 
remain proficient and current in the acquisition workforce. The 36 tasks of the SMCT 
serve to train the CCO in practical application of his craft so that he may be able to 
perform his mission while deployed. Finally, the contracting tasks the CCO performs 
while deployed round out the multitude of requirements a CCO must accomplish to be 
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proficient. The proficiency-assessment test developed in this research will serve as a 
complement to the SMCT in this model, providing feedback to the CCO and his leaders 
on his ability to accomplish the tasks. 


Warrior Task Training 


36 Contracting Tasks 
In Contracting Offices 
per the SMCTs 


Balance Many 
Requirements 


Contracting Tasks 
Performed During 
Contingency 




Civilian 

Education 


Mandatory Military 
Training 


Figure 9. CCO Development Model 
(ECC, 2009, September 22) 

CCOs assigned to the ECC may come from a multitude of acquisition or 
operational Army experiences. They all, however, must meet the minimum education 
and training requirements prior to being assigned to the ECC as a CCO and must 
continue to train in preparation for deployment. The increased contingency-contracting 
requirements and the growing number of CCOs within the ECC, a relatively new 
organization, means there are shortages of CCOs deployed in support of contingency 
operations. This assures that almost every newly assigned CCO arriving to a subordinate 
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unit within the ECC will have an opportunity to deploy. Figure 10, from the ECC’s 
briefing on phased development, depicts the typical timeline that CCOs currently 
experience from arrival to their unit through their first deployment and beyond. After 
being assessed into the Army Acquisition Corps (AAC) as a 51C, the CCO receives the 
preliminary functional training, awarding their Defense Acquisition Workforce 
Improvement Act (DAWIA) Contracting Level I. Upon arrival to a team, the CCO 
continues to train for deployment, under the direction of a team leader or battalion 
commander, on the various requirements listed in Figure 9. It is during this time that a 
CCO may first take the PAT so that he or his leaders can assess his proficiency in the 36 
tasks. From here, the team leader or battalion commander can work with the CCO to 
develop a specialized training program to improve his proficiency. After several months 
of training, the CCO may take the PAT again to assess proficiency once more. 

These follow-on PATs may be used to gauge the CCO’s progress and the 
effectiveness of a commander’s training program. After a minimum of one year of 
training at their duty station, the CCO may deploy in support of a contingency operation. 
One of the recommendations of the Gansler Commission that was adopted by the Army 
was to not send an untrained CCO into theater (Gansler, 2007). This standard timeframe 
of allowing a CCO to train for up to one year prior to deployment was adapted by the 
Army and the ECC and is depicted in Figure 10. 


55 



Apr 13 


, 0 ^ . 

Qfy February 11 


May 11 




May 12-Mar 13 


Second 
Operational 

CCO And De P lo > ment 
Professional 



+ 6 to 10 mo 
July 10 


+ 2 to 3 mo 
May 10 


+ 3 to 6 mo 
March 10 


Arrive On 
Team 


2 Month 
Exercise 
Mission 
Support 


Operational 

Deployment 


CON 234 


Back To Developmeilt 
t Continues 

Team 


School 
DAULVI 
Assessed Functional 

Training Minimum 


• NCO will need DAU LVII Training and Army Acq. 
Intermediate Contracting Course (ANCOC) 

• MSG goes to Intermediate Qualification Course 

• Officer may need Intermediate Level Education 

►NCOs Army Acquisition Basic Course (BNCOC) or Ready Airman Course 


Figure 10. CCO Development Timeline 
(ECC, 2009, September 22) 


E. CONTRACT MANAGEMENT OVERVIEW 

The contract-management process from the buyer’s perspective can be broken up 
into six steps. Garrett (2007) identifies these phases as (1) Procurement Planning, (2) 
Solicitation Planning, (3) Solicitation, (4) Source Selection, (5) Contract Administration, 
and (6) Contract Closeout or Termination. Rendon & Snider (2008) describe each of the 
six steps in the contract-management process as follows: Procurement Planning involves 
identifying what requirement can be met by procuring products or services from outside 
the organization. This phase involves whether to procure, what and how much to 
procure, and when to procure it. Solicitation Planning involves the preparation of the 
documents needed to support the solicitation. Solicitation is the process of actually 
obtaining the bids and proposals from the prospective sellers. Source Selection is the 
process of receiving and evaluating the bids and proposals from prospective offerors and 
determining which offeror to select. Contract Administration is the process of ensuring 
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that each of the parties involved in the contract meet their respective contractual 
obligations. Contract Closeout or Termination signifies the end of the contractual 
obligation between buyer and seller. It also involves ensuring that all administrative 
obligations stemming from the contract are complete (Rendon & Snider, 2008). 

These phases logically follow the lifecycle of a contract from start to finish. The 
SMCT is organized in a similar fashion. While not specifically aligned with the six 
phases of the contract management process as identified by Garrett, the subject area 
codes in the SMCT follow the natural progression of the lifecycle of a contract. Subject 
area code 1 is contracting customer support tasks; subject area code 2 is contracting pre¬ 
award tasks; subject area code 3 is contracting post-award tasks; and subject area code 4 
is other contracting officer tasks. 

F. SMCT 

The Soldier’s Manual of Common Tasks identifies training requirements for 
soldiers, supervisors, trainers, and commanders in career-management field 51C. It is 
used by commanders to plan, conduct, and evaluate individual training (DoA, 2009, 
September 22). The Army training strategy relies on soldiers, noncommissioned officers, 
and officers to be trained and ready to fight our nation’s wars. This dictum demands that 
soldiers and their commanders continuously plan and conduct realistic training. The 
SMCT provides the framework of that training for CCOs. The 36 training tasks that 
comprise the body of knowledge for a CCO are broken down into four subject area codes. 
The four subject codes are as follows: 

1. Contracting Customer Support Tasks 

2. Contracting Pre-award Tasks 

3. Contracting Post-award Tasks 

4. Other Contracting Officer Tasks 

(DoA, 2009, September 22) 

Each subject code, in turn, has corresponding tasks. Figure 11 depicts the 36 SMCT 
tasks. Subject codes are not in any particular order of difficulty or skill. All CCOs, 
regardless of rank, should be proficient in all tasks, regardless of subject code. 
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Subject 

Area 

Task 

Number 

Task 

Subject 

Area 

Task 

Number 

Task 

1 

1-1 

Identify Supported Units 

3 

3-4 

Exercise Options 

1 

1-2 

Train Supported Units (Regarding Contracting 
Proceedures) 

3 

3-5 

Monitor Contract Performance 

3 

3-6 

Process Documents For Payment 

1 

1-3 

Advise Supported Units (Regarding Contracting Matters) 

3 

3-7 

Modify Contracts 

3 

3-8 

Terminate Contracts 

1 

1-4 

Prepare Contracting Support Integration Plans 

3 

3-9 

Conduct Contract Closeout 

2 

2-1 

Review Purchase Requests and Supporting Documents 

3 

3-10 

Process Protests 

2 

2-2 

Conduct Market Research 

3 

3-11 

Process Claims 

2 

2-3 

Conduct Acquisition Planning 

3 

3-12 

Process Freedom Of Information Act Requests 

2 

2-4 

Document Other Than Full and Open Competition 

3 

3-13 

Process Unauthorized Commitment s 

2 

2-5 

Train, Designate, Monitor And Terminate Contracting 
Officer Representatives (CORs) 

4 

4-1 

Use Standard Form 44s (SF 44) 

4 

4-2 

Train. Designate. Monitor And Terminate a Field 
Ordering Officer (FOO) 

2 

2-6 

Train, Designate Monitor and Terminate Ordering Officers 
(OO) 

2 

2-7 

Synopsize Proposed Contract Actions 

4 

4-3 

Issue Letter Contracts 

2 

2-8 

Solicit Competition 

4 

4-4 

Conduct Vendor Education 

2 

2-9 

Receive Solicitation Responses 

4 

4-5 

Employ The Government Purchase Card (GPC) As 
Payment Method and Procurement Tool 

2 

2-10 

Evaluate Offers 

2 

2-11 

Prepare Contract Awards 

4 

4-6 

Manage Bulk Funds 

3 

3-1 

Conduct Post Award Orientations 

4 

4-7 

Prepare For Deployment 

3 

3-2 

Notify Unsuccessful Offerors-Award Synopsis 

4 

4-8 

Maintain PIIN Logs 

3 

3-3 

Issue Task Orders/Delivery Orders or BPA Calls 



Figure 11. 36 MOS 51 Common Tasks (SMCT) 

(ECC, 2009, September 22) 

Each of the 36 tasks is broken down into conditions, standards, and performance 
steps. The conditions of the task identify “any equipment, tools, references, job aids, and 
supporting personnel the Soldier needs to use to perform the task in wartime. It also 
identifies any environmental conditions that can alter task performance such as visibility, 
temperature or wind” (DoA, 2009, September 22). The standards are normally described 
in terms of accuracy, completeness, and speed and serve to describe how well and to 
what level the task must be performed during wartime. The performance steps comprise 
the training and evaluation section. These steps identify specific actions that a Soldier 
must do to successfully complete the task. In terms of developing the PAT, the 
conditions will help develop the scenarios, and the performance steps will help develop 
the answers to the test. 
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G. SUMMARY 

This chapter described in detail the Expeditionary Contracting Command and the 
Soldier’s Manual of Common Tasks. The chapter also described the conditions under 
which the ECC was established, its organizational structure, and its purpose. Moreover, 
it broke down the subordinate units within the ECC, from Headquarters to individual 
CCO. Next, the chapter described in detail the multitude of training requirements for 
CCOs within the framework of the ECC’s training methodology. Finally, it broke down 
the individual subcomponents of the SMCT. 

The ECC was established in short order and has put in to place many systems to 
improve the numerous shortfalls identified in the Gansler Commission’s report and other 
governmental oversight reports. It is striving to train and develop 400 newly assigned 
uniformed CCOs and 1,000 civilian contracting personnel. In doing so, it must also 
maintain command and control of current CCOs. It has developed a model training plan 
for CCOs and provided the training doctrine in the form of the Soldier’s Manual of 
Common Tasks to support their goals. However, the SMCT remains incomplete without a 
formalized proficiency-assessment test to measure a CCO’s performance against the 
SMCT standards. The purpose of this research is to develop such a test. Chapter IV, 
PAT Development, will cover in detail the process the research team used to develop the 
PAT. Chapter IV also describes in detail how the proficiency assessment test was 
created. It describes the research team’s use of the test-construction theoretical 
framework and accounts for any deviations made in that framework due to time or 
academic constraints. 
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IV. PAT DEVELOPMENT 


A. INTRODUCTION 

This chapter discusses the steps and methodology the research team took in 
developing the PAT. In this chapter, we explain how we applied the theoretical academic 
principles of test construction we learned during our literature review and how we 
applied them given our limitations, both as students (non-professional test writers) and in 
terms of time constraints, given our academic schedule. 

B. TEST CONSTRUCTION PLAN 

The development of the PAT was planned with the test-user and test-taker in 
mind. The research team had to build the PAT to meet the needs of the Expeditionary 
Contracting Command and the CCOs who would eventually take the test. We envisioned 
the PAT being taken in a digitized format, as this would be the most efficient manner in 
both administering the test, and in taking it. Moreover, a digitized version of the test 
would be less susceptible to compromise by would-be test takers, and would lend itself 
better to modifications. While placing the PAT in its final digitized version was not part 
of the scope of this project, the questions and answers written by the research team had to 
be developed in a manner that would make a transfer to digitized form convenient. 

In developing the PAT, the research team could not follow the steps to test 
construction identified in the literature review as describe by either Downing & Haladyna 
(2006) or Allen & Yen (2001). We knew early on that our academic calendar and 
limited knowledge in test construction would not allow us to develop the test in this 
manner. Instead, we prioritized the major steps that were common between the two lists 
and focused on test blueprinting and item writing. 

C. TEST BLUEPRINT 

The research team first developed a test blueprint for each of the tasks. The test 
blueprint acted as a road map for test construction by laying out which subject areas we 
wanted to test and at what cognitive domains the subject area should be tested. This test 


61 



blueprint broke down each of the tasks into subcomponents called items. Each item is a 
specific subset of knowledge of that particular task. The research team developed a 
spreadsheet that listed each of the items within a specific task with several questions 
corresponding to each of item. We then wrote questions that tested knowledge of that 
item. As we wrote questions for each task, we annotated on the spreadsheet which 
cognitive domain level the questions tested. In this way, we were able to determine when 
we had adequate numbers of questions for each item that tested at each cognitive domain 
category. 

D. ITEM WRITING 

The research team took each task and broke it down to its subcomponents of task, 
conditions, and performance steps. The tasks in the SMCT represent the 36 testable tasks 
that we used to measure proficiency. Having the task, conditions and performance test 
essentially gave us the questions and answers. We used these elements of the tasks of the 
SMCT to construct the test questions. 

The research team used the knowledge gained during the literature review and 
planned the PAT as a criterion-referenced test. Criterion-referenced tests attempt to 
determine if an individual has mastered a subject area in a specific learning objective 
(Marshall & Hales, 1972). In the case of the PAT, this test would attempt to determine of 
CCOs had mastered the particular task within the SMCT. 

E. VALIDITY AND RELIABILITY 

Test validity and reliability were two essential test characteristics we identified 
during our literature review that we had to pay attention to during item writing. The 
research team, as much as possible, wanted to ensure the PAT was written with these two 
attributes in mind. We accepted that we would not be able to measure either validity or 
reliability in our test, as doing so would involve issuing the test to sample populations 
and measuring consistency in responses (Allen & Yen, 2001; APA, 1974). The academic 
and time constraints would not allow us to follow through in the test development in this 
manner. However, we did take into account factors to maximize both validity and 
reliability in our test. 
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To ensure reliability, the research team incorporated guidelines for item writing as 
presented by Kline (1986). These guidelines included (1) reducing the insight which 
subjects may have into the items, (2) writing in clear, unambiguous terms, (3) referring to 
specific instead of general behavior, (4) ensuring each item asked only one question or 
made one statement, (5) and avoiding as much as possible terms of frequency (Kline, 
1986). 

Additionally, we looked at test length to increase reliability. Generally, the longer 
the test, the more reliable it is (Nunnally & Bernstein, 1978). Based on this, the research 
team attempted, whenever possible, to write at least twenty items for each task. The 
length of the task, conditions and performance measures for each of the tasks varied and 
as a result, so did the number of items for each task. Some tasks required well over thirty 
questions, while some were so short that only ten could be developed. 

In an effort to prevent compromise of the test by would be test takers, the research 
team doubled the number of questions for each task from an average goal of twenty 
questions per task to an average goal of forty questions per task. This doubling of items 
would serve to increase the database of questions for a digitized version of the test, thus 
allowing for multiple versions of the test to be administered. In addition, the doubling of 
items would result in the greater validity. 

The research team also took in to account efforts in the development of the PAT 

to ensure content validity of the test. According to Nunnally (1972), validity is how well 

a test meets its designed function. The PAT was designed to measure proficiency in the 

36 tasks listed in the SMCT. One of the ways that the research team ensured the test 

measured the content of those tasks was by deconstructing the task, conditions, and 

performance measures in the SMCT and rewriting them as questions. For example, for 

Task 1-1, Identify Supported Units , one of the performance steps is to recognize key 

personnel and support elements. The SMCT lists several key positions the CCO may 

come in contact with. Under the position of Finance Officer, it states, “Key element 

responsible for banking and currency support, disbursing functions, and electronic funds 

transfers.” In order to ensure that the question measured the content of the performance 

step, the research team would rephrase the verbiage in the SMCT in the form of a 
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question as such: “Who is responsible for banking and currency support, disbursing 
functions, and electronic funds transfers?” This question would be followed by several 
possible answers, one of which would be the correct answer (Finance Officer). The other 
(incorrect) answers would be the other ones from the list of the other key personnel 
described in the SMCT. 

Once the questions for each subject area of questions were completed, they were 
e-mailed, in a Word document, to our sponsor at the Expeditionary Contracting 
Command. Our sponsor would then mail these questions out to senior civilian and 
military CCOs for vetting. This vetting process by senior leaders and experts in the field 
of contingency contracting was another way to ensure content validity (Nunnally & 
Bernstein, 1978). 

F. COGNITIVE DOMAINS 

The research team took in to account the application and use of the test when 
determining the cognitive domains that the test would measure. The ECC needed the 
PAT to administer to thousands of CCOs throughout the Army. This necessitated a test 
that could be administered to numerous individuals, at different times and locations, by 
numerous test administrators. Because of this, the final version of the test would need to 
be computer-based. For this reason, most all of the questions and answers were written 
in a multiple-choice format. As such, most of the questions written tested knowledge at 
the remember and understand categories, and only to a limited extent, application 
categories of Bloom’s cognitive domains. 

As previously stated, most all of the questions were written as what are commonly 
referred to as multiple-choice questions. These multiple-choice questions, known as 
selected-response items, occur when the examinee is given both the problem and the 
correct solution and then must choose the correct solution over the incorrect solutions. 
Selected-response items were chosen over-constructive response type questions for the 
PAT since selected-response were less resource intensive and more conducive to a 
digitized format. The research team expanded the format of selected-response questions 
to include true/false, matching and sentence completion. 
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Wood (1960) suggests item writers should establish a random system to position 
the correct response when developing selective-response items in order to avoid bias 
within the test. The reasoning for this is that test developers will inadvertently choose a 
particular response position over others and this may induce bias in the test. In other 
words, a test developer could inadvertently end up with the response letter A, for 
example, as the answer to most of the multiple-choice questions. In order to mitigate 
this, every question the research team wrote had the first response, or the answer letter A, 
as the correct answer. In this way, the ECC or future test developers who transfer the test 
to a digitized usable form could randomly shuffle the available choices and the correct 
response. Additionally, this allowed for ease of vetting by the ECC for validity since it 
eliminated the need for an answer key. 

G. SUMMARY 

Over the course of four months, the research team wrote approximately forty test 
questions for each of the 36 tasks in the SMCT. The tasks varied in length so some of the 
tasks had as few as 10 questions and some had as many as 60. In total, the research team 
wrote over 1,100 questions for the SMCT spanning over 300 pages of text. Many of the 
questions were developed to test redundant items. In this way, these questions could be 
used in different versions of the test. The question and answer sheets were submitted to 
our sponsor at the ECC in Word format. At the time of this writing, the ECC and their 
senior level leaders of experienced CCOs had received all questions for the 36 tasks the 
team developed. The next chapter is the conclusion chapter of this research project. It 
lists recommendations for administration of the test and suggests areas of further 
research. 
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V. SUMMARY, CONCLUSION, AND AREAS FOR FURTHER 

RESEARCH 


A. SUMMARY 

U.S. Army contingency contracting officers (CCOs) have played a critical role in 
nearly every overseas military operation by providing supplies and services to the 
warfighter in an efficient, timely, and ethical manner. Despite this critical role, the Army 
has not integrated its CCOs or the procurement process in a holistic approach to its 
expeditionary, warfighting mindset. Shortfalls in the number of U.S. Army CCO 
personnel and the lack of a standardized training process have negatively affected the 
Army’s expeditionary procurement capabilities. 

The U.S. Army’s Expeditionary Contracting Command (ECC) was established in 
January 2008 in direct response to the Gansler Commission’s recommendations to 
prepare, train, and provide skilled contracting personnel in support of U.S. Army 
contingency operations. This new command, and other resultant changes of the Gansler 
Commission’s recommendations, resulted in a planned net increase of over 400 
uniformed and 800 civilian contracting officer positions. The ECC developed a 
standardized set of 36 tasks to properly train CCOs prior to deployment. These 36 tasks, 
published in the ECC’s Soldier’s Manual of Common Tasks (SMCT) MOS 51C 
Acquisition, Logistics, and Technology, represent the body of knowledge that a 51C 
CCO must know to be successful during contingency operations. 

The establishment and standardization of training tasks by the ECC for CCOs in 
the MOS 51C SMCT, in accordance with Army training doctrine, has helped subordinate 
units and individual CCOs within the ECC to focus their training efforts prior to 
deployment. This body of knowledge is a significant step in standardizing the pre¬ 
deployment training for CCOs and preparing them for their mission once deployed. To 
supplement this training, an appropriate performance-assessment tool is needed to 
measure performance of these 36 tasks. Currently, the ECC lacks a standardized method 
of measuring individual skill and task proficiency in its CCOs. 
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The written proficiency-assessment test questions and answers developed in this 
research will allow leaders in the ECC to measure a CCO’s knowledge, skills, and 
capabilities. A PAT will enable leaders to measure individual task proficiency against 
established standards, determine each CCO’s preparedness to conduct duties during 
deployment, make appropriate CCO assignments, and develop individualized training 
programs for CCOs. 

This research developed the questions for a written proficiency test based on the 
36 tasks listed in the SMCT. The research team developed the scenarios and questions to 
assess proficiency in accordance with conditions, standards, and performance steps for 
each of the 36 tasks in the SMCT. Questions were structured to assess Bloom’s cognitive 
domains of remembering, understanding, and, to a limited extent, application of tasks. 
The research team vetted each of the scenarios and accompanying questions through 
experienced CCO focus groups within the ECC. Upon completion of the test, the 
research team provided a written Contracting Officer Proficiency-Assessment Test, 
encompassing over 1,100 questions covering each of the 36 tasks listed in the SMCT, to 
the Expeditionary Contracting Command. 

B. CONCLUSION 

The results of this research project, over 300 pages containing over 1,100 multiple 
choice and their respective answers, was developed over the course of a year. These 
documents represent the first step in developing a proficiency assessment test for the 
Expeditionary Contracting Command’s growing ranks of CCOs. 

At the beginning of this research project, we posed the following research 
question: What is the most effective and efficient method of measuring Individual CCO 
task proficiency in the 36 tasks listed in the SMCT ? In answering this question, we found 
that a multiple choice (selective-response) type test administered in digitized format 
directly measuring the performance measures in the SMCT would be the answer to this 
question. In developing the PAT, we never intended to convert the final PAT into a 
digitized version, as this was outside the scope of our project. We did, however, write the 
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test questions and answers that would go into such a test, and we formatted those 
questions and answers in such a way that could easily be converted into a digitized test by 
follow-on researchers. 

Secondary research questions we asked and attempted to answer were as follows: 

• How can a proficiency-assessment test be developed? In answering this 
question, we looked to core test principles, test construction methods, test 
blueprints and item writing covered in our literature review for the answer. 
Instead of providing the answer to this question, we used the knowledge 
gained in answering the question to develop the PAT questions. 

• What type(s) of testing is best suited to evaluate the cognitive domains of 
remembering, understanding, and, to a limited extent, application of 
knowledge? As in the previous secondary research question, the answer to 
this question helped the research team formulate the best type of test given 
the requirements of the ECC. Given that any final version of the PAT 
would have to administered at various locations throughout the world to 
numerous CCOs at varying times, selective-response were deemed best 
suited. 

• What type(s) of testing is most efficient—in terms of requiring the least 
amount of resources, such as overhead and setup—for the tester and test 
taker? This last question identifies the need for the PAT to be 
administered, taken, and graded in a digitized format. 

The answers to the secondary research questions were discovered by conducting 
our literature review on test construction and in developing the test. We asked and 
attempted to answer these questions not for the answers in and of themselves, but to clue 
us in into how to make the most effective proficiency-assessment test for the ECC. In 
this sense, answering these research questions did not, by themselves, satisfy the 
requirements of the research project. Instead, answering these research questions gave us 
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the know-how in how to write the PAT. The result of the knowledge gained in answering 
these questions is the over 1,100 questions written by the research team and presented to 
the ECC for inclusion the PAT. 

At the conclusion of this research project, all question and answers were 
submitted to the ECC for review. This transferring of questions and answers signifies 
transfer of ownership of this body of knowledge that was created during the conduct of 
this research, from the research team, to the Expeditionary Contracting Command. The 
ECC can now use these questions to complete the conversion of the PAT from its current 
raw form to a usable and implementable test. 

C. AREAS FOR FURTHER RESEARCH 

There are several areas the research team recommends for follow-on research. 
This research was able to develop the questions and answers for a PAT for the SMCT. 
However, the test as presented in this research project is not in a usable form. Most of 
our recommendations for further research are tied to the continued development of the 
PAT as a usable test for the ECC. Specifically, the research team makes the following 
recommendations: (1) Conversion of the test from its current, written form to a digitized 
version, (2) Measure the test for validity and reliability, (3) Increase the number of 
questions in the question and answer database, (4) Increase the number of scenario- and 
performance-based questions that measure at the application category cognitive domain 
and (5) Continually validate the 36 tasks listed in the SMCT as current given emerging 
trends in contingency-contracting. 

The test, in its present form cannot be administered. The product produced by 
this research project resulted in over 1,100 questions with multiple choice answers, the 
correct answer for each being the first one, or A. A follow-on group of researchers could 
take the questions and answers this research has developed and convert them into 
digitized version that could be administered, taken, and graded via computer. This 
computerized test could serve as the prototype test for the ECC from which the test could 
be measured for reliability and validity, if the ECC desired. 
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The test should be measured for validity and reliability. While we took steps to 
mitigate the test development errors that affect validity and reliability, we could not fully 
validate the test as valid and reliable. A follow on research team could issue the test in a 
digitized form to a sample pool of CCOs and measure if in fact the test is an accurate 
measure of the knowledge it is intended to measure (validity) and that it continuously 
does so (reliability). 

The test should include additional questions in multiple combinations to create 
various versions of the test. When we designed the test, we attempted to write at least 
twenty questions per task. In an effort to limit test memorization and compromise, we 
increased the number of test questions to forty per task for each of the 36 tasks. These 
questions may have to be replaced during the vetting process, or when the test is 
measured for validity and or reliability. These omitted questions should be replaced, and 
new questions should be added to increase the number of questions in the question 
database. 

Additionally, questions that measure more demanding cognitive domain 
categories such as application of knowledge should be developed for the test. This test, 
while computer-administered and taken, could still include performance-based questions 
where test takers would have to read a scenario and fill out proper forms related to 
contingency-contracting. This research effort would not only increase the quantity of 
questions, but also more accurately measure a CCO’s abilities to perform their mission by 
testing at a more demanding cognitive level. 

The tasks contained in the SMCT should be validated against emerging trends in 
contingency contracting. This PAT was developed based on the 36 tasks in the SMCT. 
Our literature review showed that tasks in contingency-contracting training changed as 
the nature of warfare has changed. Based on this, the research team assumes that the 36 
tasks in the SMCT are also susceptible obsolescence. A periodic review of the relevance 
of the 36 tasks contained in the SMCT would assist test administrators and developers of 
the PAT to refine both the SMCT and the content of the PAT. 
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Finally, the research team acknowledged early on that we would not be able to 
fully develop the PAT into its final, usable form as a computer administered test. To this 
end, we have sought to establish a long-term partnership between the Naval Postgraduate 
School (NPS) and the Expeditionary Contracting Command to continue to develop and 
refine the proficiency-assessment test for CCOs. The ECC should continue to reach out 
to the NPS to facilitate the evolution of the test. Similarly, NPS faculty and students 
interested in the field of study of contingency contracting should look to ECC for real- 
world problem sets with which to assist. 

The proficiency-assessment test as presented in this research is a giant step in 
standardizing an individual proficiency assessment tool. While it is not immediately 
usable and implementable by the ECC in its current form, it is our hope that the products 
resulting from this research will serve the ECC as solid base in furtherance of this effort. 
Once implemented, the proficiency-assessment test submitted as part of this research will 
allow leaders in the ECC to measure a CCO’s knowledge, skills, and capabilities by 
measuring individual task proficiency against the established standards contained in the 
SMCT. The PAT will finally allow leaders within the ECC to effectively and efficiently 
determine each CCO’s preparedness to conduct duties during deployment, make 
appropriate CCO assignments, and develop individualized training programs for CCOs. 
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